# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=282 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 283 --- ## [How to schedule logstash pipelibe to consume kafka input logs every 15min](https://discuss.elastic.co/t/how-to-schedule-logstash-pipelibe-to-consume-kafka-input-logs-every-15min/255320)
**Author:** [@soufian.eldouqe](https://discuss.elastic.co/u/soufian.eldouqe)\ **Replies:** 0\ **Last updated:** [November 13, 2020, 10:04am UTC](https://discuss.elastic.co/t/how-to-schedule-logstash-pipelibe-to-consume-kafka-input-logs-every-15min/255320 "2020-11-13T10:04:13Z")
Hello, I want to developp a logstash pipeline that consume heathbeat logs from filebeat that are generated every 5 minutes. the goal is to be sure that everything is working fine from my filebeat inputs to kibana. how … --- ## [How to express SQL in Logstash](https://discuss.elastic.co/t/how-to-express-sql-in-logstash/255049)
**Author:** [@tusharnemade](https://discuss.elastic.co/u/tusharnemade)\ **Replies:** 1\ **Last updated:** [November 13, 2020, 8:08am UTC](https://discuss.elastic.co/t/how-to-express-sql-in-logstash/255049 "2020-11-13T08:08:13Z")
Hello Team I am using Logstash and Elasticsearch version 7.8.0. I want to write query in logstash for input plugin of elasticsearch as select userId from myindex I am unable to achieve same in logstash ... --- ## [Can't change port for logstash](https://discuss.elastic.co/t/cant-change-port-for-logstash/255299)
**Author:** [@Hi\_u\_Nguy\_n\_Dang](https://discuss.elastic.co/u/Hi_u_Nguy_n_Dang)\ **Replies:** 0\ **Last updated:** [November 13, 2020, 7:18am UTC](https://discuss.elastic.co/t/cant-change-port-for-logstash/255299 "2020-11-13T07:18:33Z")
How can I change from port 5044 to 4000 for filebeat/logstash guys, it show this error Tks in advance --- ## [Ingest xml file using Logstash](https://discuss.elastic.co/t/ingest-xml-file-using-logstash/255113)
**Author:** [@Priyanka3](https://discuss.elastic.co/u/Priyanka3)\ **Replies:** 9\ **Last updated:** [November 12, 2020, 8:29pm UTC](https://discuss.elastic.co/t/ingest-xml-file-using-logstash/255113 "2020-11-12T20:29:55Z")
Hi, I am very new to Elastic search and Logstash. But I am trying to ingest using Logstash. My xml file looks something like this \ **Author:** [@Kiwito](https://discuss.elastic.co/u/Kiwito)\ **Replies:** 3\ **Last updated:** [November 12, 2020, 11:49pm UTC](https://discuss.elastic.co/t/fatal-error-block-in-setup-after-succesfull-connection/255255 "2020-11-12T23:49:24Z") Hi there, I have been trying to solve this problem for a day. I couldn't find any solutions. \[2020-11-12T20:07:47,943\]\[INFO \]\[logstash.filters.geoip \]\[main\] Using geoip database {:path=\>"/opt/logstash/vendor/geoi… --- ## [How to convert strings to time format](https://discuss.elastic.co/t/how-to-convert-strings-to-time-format/255267)
**Author:** [@Stephenson](https://discuss.elastic.co/u/Stephenson)\ **Replies:** 0\ **Last updated:** [November 12, 2020, 11:01pm UTC](https://discuss.elastic.co/t/how-to-convert-strings-to-time-format/255267 "2020-11-12T23:01:31Z")
Hello everyone, Since a few months, I started working with logstash to small programs like reading a log. I have some trouble understanding the development. I have this line of log. 08:19:1915 , Error trae\_1dato I… --- ## [Undefined method \`get' for](https://discuss.elastic.co/t/undefined-method-get-for/255214)
**Author:** [@ankitdevnalkar](https://discuss.elastic.co/u/ankitdevnalkar)\ **Replies:** 12\ **Last updated:** [November 12, 2020, 4:32pm UTC](https://discuss.elastic.co/t/undefined-method-get-for/255214 "2020-11-12T16:32:47Z")
I am facing bellow error while using http\_poller. I am trying to pull cloudflare API logs in json format. Here is my .conf file. I am using logstash 7.9.2 and logstash-input-http\_poller (5.0.2) Can anyone please help ? … --- ## [How to read data from a DB table and project it on KIBANA DB](https://discuss.elastic.co/t/how-to-read-data-from-a-db-table-and-project-it-on-kibana-db/254420)
**Author:** [@Dinesh\_Potey](https://discuss.elastic.co/u/Dinesh_Potey)\ **Replies:** 9\ **Last updated:** [November 12, 2020, 3:31pm UTC](https://discuss.elastic.co/t/how-to-read-data-from-a-db-table-and-project-it-on-kibana-db/254420 "2020-11-12T15:31:27Z")
Hi, I want to run a select query on a DB table so that I project the output on KIBANA DB. I am not expertise in modules. Could you please help me here in achieving this. Thanks, Dinesh --- ## [Logstash output log messages to multiple outputs simultaneously?](https://discuss.elastic.co/t/logstash-output-log-messages-to-multiple-outputs-simultaneously/255232)
**Author:** [@randoran](https://discuss.elastic.co/u/randoran)\ **Replies:** 1\ **Last updated:** [November 12, 2020, 3:29pm UTC](https://discuss.elastic.co/t/logstash-output-log-messages-to-multiple-outputs-simultaneously/255232 "2020-11-12T15:29:40Z")
Is it possible for logstash to output a log message to multiple different outputs simultaneously? Something like this: output { if \[env\] == "production" and \[node\] == "web101" { file { path =\> "/var/log/logs… --- ## [Logstash - Filter events according to data from a REST API](https://discuss.elastic.co/t/logstash-filter-events-according-to-data-from-a-rest-api/255153)
**Author:** [@Yor\_On](https://discuss.elastic.co/u/Yor_On)\ **Replies:** 1\ **Last updated:** [November 12, 2020, 3:16pm UTC](https://discuss.elastic.co/t/logstash-filter-events-according-to-data-from-a-rest-api/255153 "2020-11-12T15:16:54Z")
Hi, I've scoured the internet for someone with the same issue as I, but I haven't had much success in finding any precedent on getting what I need done. I have a REST API that returns a response which looks like: GET … --- ## [Logstash A plugin had an unrecoverable error (AWS ARM64/Ubuntu 20.04/Java OpenJDK 11)](https://discuss.elastic.co/t/logstash-a-plugin-had-an-unrecoverable-error-aws-arm64-ubuntu-20-04-java-openjdk-11/255146)
**Author:** [@yoesoff](https://discuss.elastic.co/u/yoesoff)\ **Replies:** 1\ **Last updated:** [November 12, 2020, 2:53pm UTC](https://discuss.elastic.co/t/logstash-a-plugin-had-an-unrecoverable-error-aws-arm64-ubuntu-20-04-java-openjdk-11/255146 "2020-11-12T14:53:15Z")
I have installed ELK using docker from following link https://github.com/deviantony/docker-elk/tree/release-7.x It works very well before, but recenty I got an issue on several new servers. The issue related with Logsta… --- ## [Use logstash or filebeat or both to send log at S3](https://discuss.elastic.co/t/use-logstash-or-filebeat-or-both-to-send-log-at-s3/255223)
**Author:** [@Falikou1](https://discuss.elastic.co/u/Falikou1)\ **Replies:** 0\ **Last updated:** [November 12, 2020, 2:23pm UTC](https://discuss.elastic.co/t/use-logstash-or-filebeat-or-both-to-send-log-at-s3/255223 "2020-11-12T14:23:55Z")
I want all logs that are sent to logstash or filebeat, to be forwarded to S3. I have a question? Should I use logstash or filebeat or both? Example configuration: If it's logstash If it's filebeat If both --- ## [Getting error while pulling data from an API in Logstash](https://discuss.elastic.co/t/getting-error-while-pulling-data-from-an-api-in-logstash/254779)
**Author:** [@ankitdevnalkar](https://discuss.elastic.co/u/ankitdevnalkar)\ **Replies:** 5\ **Last updated:** [November 12, 2020, 12:20pm UTC](https://discuss.elastic.co/t/getting-error-while-pulling-data-from-an-api-in-logstash/254779 "2020-11-12T12:20:43Z")
Hey there, can any help me resolve this problem, I am parsing Cloudflare audit logs via its API, I am putting here my code chunk of .conf file for Logstash plugin input,filter and output. Also, attaching a error screen… --- ## [Logstash error trying to index filebeat](https://discuss.elastic.co/t/logstash-error-trying-to-index-filebeat/255175)
**Author:** [@Craig2188](https://discuss.elastic.co/u/Craig2188)\ **Replies:** 1\ **Last updated:** [November 12, 2020, 9:32am UTC](https://discuss.elastic.co/t/logstash-error-trying-to-index-filebeat/255175 "2020-11-12T09:32:18Z")
Hi, Logstash log files are being filled up with indexing errors for filebeat: \[2020-11-12T09:06:02,844\]\[WARN \]\[logstash.outputs.elasticsearch\]\[main\]\[28a01e4629fbbc3006822900269029f08bd4f371d6be029377ac4944518e3137\] Cou… --- ## [Split json in string from packetbeat](https://discuss.elastic.co/t/split-json-in-string-from-packetbeat/254874)
**Author:** [@langol93](https://discuss.elastic.co/u/langol93)\ **Replies:** 2\ **Last updated:** [November 12, 2020, 9:22am UTC](https://discuss.elastic.co/t/split-json-in-string-from-packetbeat/254874 "2020-11-12T09:22:19Z")
Hello I passed http request from packetbeat to logstash and now I have json in htttp.request.body.content: How can I split this string to separate fields? I want to have id, timestamp and value field.. I tried usi… --- ## [Logstash Error after setting up TLS on Elastic](https://discuss.elastic.co/t/logstash-error-after-setting-up-tls-on-elastic/255055)
**Author:** [@Craig2188](https://discuss.elastic.co/u/Craig2188)\ **Replies:** 3\ **Last updated:** [November 12, 2020, 9:09am UTC](https://discuss.elastic.co/t/logstash-error-after-setting-up-tls-on-elastic/255055 "2020-11-12T09:09:36Z")
I have enabled security on Elastic and setup HTTP/TLS for communication between Elastic and Kibana. Elastic and Kibana are now up and working fine with those new changes. I have just setup the following for Logstash for… --- ## [Reading csv file from dropbox](https://discuss.elastic.co/t/reading-csv-file-from-dropbox/255170)
**Author:** [@byteandbit](https://discuss.elastic.co/u/byteandbit)\ **Replies:** 0\ **Last updated:** [November 12, 2020, 8:37am UTC](https://discuss.elastic.co/t/reading-csv-file-from-dropbox/255170 "2020-11-12T08:37:21Z")
Is there anyway to read csv file from dropbox in logstash? --- ## [How to apply sdfc filter in salesforce plugin of logstash](https://discuss.elastic.co/t/how-to-apply-sdfc-filter-in-salesforce-plugin-of-logstash/254736)
**Author:** [@Pavitra\_Poojary](https://discuss.elastic.co/u/Pavitra_Poojary)\ **Replies:** 2\ **Last updated:** [November 12, 2020, 5:07am UTC](https://discuss.elastic.co/t/how-to-apply-sdfc-filter-in-salesforce-plugin-of-logstash/254736 "2020-11-12T05:07:56Z")
Hi Team, We are extracting the logs from Salesforce object. Here is our input configuration with a object name Error\_Log\_\_c. input {​ salesforce {​ use\_test\_sandbox =\> true client\_id =\> 'xxx' client\_secret =\> 'xxx' … --- ## [Multiple Pipeline Output](https://discuss.elastic.co/t/multiple-pipeline-output/255140)
**Author:** [@justinainsworth](https://discuss.elastic.co/u/justinainsworth)\ **Replies:** 2\ **Last updated:** [November 12, 2020, 1:05am UTC](https://discuss.elastic.co/t/multiple-pipeline-output/255140 "2020-11-12T01:05:03Z")
Continuing the discussion from Multiple outputs?: According to this post, events can pass to multiple outputs. Is that the case when output is another pipeline? Or more specifically, if all conditionals are met, can a… --- ## [Suggestions which type of filters we can use procees the logs via logstash](https://discuss.elastic.co/t/suggestions-which-type-of-filters-we-can-use-procees-the-logs-via-logstash/255141)
**Author:** [@525125](https://discuss.elastic.co/u/525125)\ **Replies:** 0\ **Last updated:** [November 12, 2020, 12:54am UTC](https://discuss.elastic.co/t/suggestions-which-type-of-filters-we-can-use-procees-the-logs-via-logstash/255141 "2020-11-12T00:54:46Z")
How to process this type of logs via logsatsh to kibana. Any suggestions on filtration. 2020-11-04 00:00:00,038 {abcd23rv6f} \[\] \[INFO \] service - response from 3rd party : \[, , , , , , authentication OK., 1, 0, SafeNet … --- ## [Logstash support for composable templates in elasticsearch output](https://discuss.elastic.co/t/logstash-support-for-composable-templates-in-elasticsearch-output/255137)
**Author:** [@phospodka](https://discuss.elastic.co/u/phospodka)\ **Replies:** 0\ **Last updated:** [November 11, 2020, 10:34pm UTC](https://discuss.elastic.co/t/logstash-support-for-composable-templates-in-elasticsearch-output/255137 "2020-11-11T22:34:57Z")
I am trying to upgrade versions of ELK and I have noticed that templates have changed. There is a legacy way and a new composable way. I currently have logstash supply custom templates for different indices. However, … --- ## [JSON to fields in Logstash](https://discuss.elastic.co/t/json-to-fields-in-logstash/255016)
**Author:** [@Stevek](https://discuss.elastic.co/u/Stevek)\ **Replies:** 6\ **Last updated:** [November 11, 2020, 3:29pm UTC](https://discuss.elastic.co/t/json-to-fields-in-logstash/255016 "2020-11-11T15:29:53Z")
Hello, I am struggling to obtain fields with values from simple JSON file using logstash. Currently putting the output to file so I see if it looks okay, but I always receive errors for each line. JSON : { "cpuLoad":… --- ## [Logstash In Windowsn not Pushing logs to kibana](https://discuss.elastic.co/t/logstash-in-windowsn-not-pushing-logs-to-kibana/255091)
**Author:** [@Rahul\_Ravichandran](https://discuss.elastic.co/u/Rahul_Ravichandran)\ **Replies:** 1\ **Last updated:** [November 11, 2020, 3:26pm UTC](https://discuss.elastic.co/t/logstash-in-windowsn-not-pushing-logs-to-kibana/255091 "2020-11-11T15:26:36Z")
Hi guys, I wanted to install logstash in windows 2019 server as my server is in AWS and my ES domain is also in AWS. Im directly using logsatsh to transfer logs to ES end point. This is my logstash.conf file i… --- ## [Parsing netflow (XML) logs, problems with encoding](https://discuss.elastic.co/t/parsing-netflow-xml-logs-problems-with-encoding/255073)
**Author:** [@th-ink](https://discuss.elastic.co/u/th-ink)\ **Replies:** 0\ **Last updated:** [November 11, 2020, 1:49pm UTC](https://discuss.elastic.co/t/parsing-netflow-xml-logs-problems-with-encoding/255073 "2020-11-11T13:49:26Z")
Hi all, I have some netflow logs in .xml that I need to parse with logstash. All is well until I reach payload field, which has differnet char set (?) and logstash crashes. This is the error message returned: Error pa… --- ## [2 hit observed in kibana whreas I ingest only once using logstash](https://discuss.elastic.co/t/2-hit-observed-in-kibana-whreas-i-ingest-only-once-using-logstash/255064)
**Author:** [@Sivajanani](https://discuss.elastic.co/u/Sivajanani)\ **Replies:** 0\ **Last updated:** [November 11, 2020, 1:02pm UTC](https://discuss.elastic.co/t/2-hit-observed-in-kibana-whreas-i-ingest-only-once-using-logstash/255064 "2020-11-11T13:02:41Z")
I have my logstash config file as below, input{ file { type =\> "json" path =\> "D:/a-json.json" start\_position =\> "beginning" sincedb\_path =\> "NUL" mode =\> "read" file\_completed\_action =\> "log" file\_comple… --- ## [Logstash idles out after a duration](https://discuss.elastic.co/t/logstash-idles-out-after-a-duration/255057)
**Author:** [@vyas.radhakrishnan](https://discuss.elastic.co/u/vyas.radhakrishnan)\ **Replies:** 0\ **Last updated:** [November 11, 2020, 12:28pm UTC](https://discuss.elastic.co/t/logstash-idles-out-after-a-duration/255057 "2020-11-11T12:28:27Z")
We have a logstash pipeline . Input is a kafka topic , output is elastic search. If the topic does not receive any messages for hours , we observe a delay in processing the next message from topic when it comes. The log… --- ## [Logstash DLQ commit\_offsets =\> true seems to be ignored?](https://discuss.elastic.co/t/logstash-dlq-commit-offsets-true-seems-to-be-ignored/255053)
**Author:** [@BBQigniter](https://discuss.elastic.co/u/BBQigniter)\ **Replies:** 0\ **Last updated:** [November 11, 2020, 12:18pm UTC](https://discuss.elastic.co/t/logstash-dlq-commit-offsets-true-seems-to-be-ignored/255053 "2020-11-11T12:18:41Z")
I'm now playing around with the dead letter queue feature of Logstash. First I tried around with commit\_offsets =\> false to be able to set up everything and as expected the DLQ events were ingested after each restart of … --- ## [Filtering specific data as per requirements from Logtash](https://discuss.elastic.co/t/filtering-specific-data-as-per-requirements-from-logtash/255052)
**Author:** [@Sandesh\_Thombare](https://discuss.elastic.co/u/Sandesh_Thombare)\ **Replies:** 0\ **Last updated:** [November 11, 2020, 12:15pm UTC](https://discuss.elastic.co/t/filtering-specific-data-as-per-requirements-from-logtash/255052 "2020-11-11T12:15:36Z")
Hi All, I am new to the logstash and trying to setup monitoring in Kibana. We have below messages & i would like to capture the message between GetInstallBase. and send it to kibana. 2020-10-28 12:10:11,149 \[fault (s… --- ## [Logstash output script not working: Append array while upserting](https://discuss.elastic.co/t/logstash-output-script-not-working-append-array-while-upserting/250824)
**Author:** [@Bastian\_Jager](https://discuss.elastic.co/u/Bastian_Jager)\ **Replies:** 4\ **Last updated:** [November 11, 2020, 10:27am UTC](https://discuss.elastic.co/t/logstash-output-script-not-working-append-array-while-upserting/250824 "2020-11-11T10:27:55Z")
My plan is to uploads logs with logstash and update a field if this is present already in ES. There is the field "origin" with maybe this “origin” : \[ “live” \] and on upload it should become “origin” : \[ “live, “upload\].… --- ## [Snmp v3 logstash not working in pipeline](https://discuss.elastic.co/t/snmp-v3-logstash-not-working-in-pipeline/254940)
**Author:** [@BenoitSERRA](https://discuss.elastic.co/u/BenoitSERRA)\ **Replies:** 5\ **Last updated:** [November 11, 2020, 9:48am UTC](https://discuss.elastic.co/t/snmp-v3-logstash-not-working-in-pipeline/254940 "2020-11-11T09:48:56Z")
Hello, I'm struggling with a strange problem with Logstash. I've created a config to query a few switches with SNMP v3 et get targeted interfaces statistics When I run logstash with the config file, everything works f… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=281) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=283)