# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=596 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 597 --- ## [RESTful APIs and http\_poller input plugin](https://discuss.elastic.co/t/restful-apis-and-http-poller-input-plugin/169450)
**Author:** [@nandrik](https://discuss.elastic.co/u/nandrik)\ **Replies:** 0\ **Last updated:** [February 21, 2019, 3:35pm UTC](https://discuss.elastic.co/t/restful-apis-and-http-poller-input-plugin/169450 "2019-02-21T15:35:55Z")
I've found myself in a few situations where there is a RESTful API that I want to consume and which updates over time with newer entries. I wonder what's the best way to be periodically probing the API and consistently … --- ## [Avoiding duplicate entries when using JDBC output to file](https://discuss.elastic.co/t/avoiding-duplicate-entries-when-using-jdbc-output-to-file/169445)
**Author:** [@wesch](https://discuss.elastic.co/u/wesch)\ **Replies:** 0\ **Last updated:** [February 21, 2019, 3:20pm UTC](https://discuss.elastic.co/t/avoiding-duplicate-entries-when-using-jdbc-output-to-file/169445 "2019-02-21T15:20:14Z")
Hi all, New to Logstash, was wondering how to go about this.. I've managed to get output to file working correctly, but would like to avoid duplicate entries. Having done some reading up, this seems to be possible using… --- ## [Problem with date filter to replace timestamp](https://discuss.elastic.co/t/problem-with-date-filter-to-replace-timestamp/169362)
**Author:** [@hackman61](https://discuss.elastic.co/u/hackman61)\ **Replies:** 6\ **Last updated:** [February 21, 2019, 3:10pm UTC](https://discuss.elastic.co/t/problem-with-date-filter-to-replace-timestamp/169362 "2019-02-21T15:10:42Z")
Hello, I have latest ELK docker image, and I send application logs from windows with filebeat, everything is almost fine :wink: I have this kinf of message : 2019-02-21 10:35:00,085 \[3140\] INFO - Début du batch EDNot… --- ## [Logstash help with geoip mapping from csv file](https://discuss.elastic.co/t/logstash-help-with-geoip-mapping-from-csv-file/169229)
**Author:** [@mmalisa12](https://discuss.elastic.co/u/mmalisa12)\ **Replies:** 4\ **Last updated:** [February 21, 2019, 3:06pm UTC](https://discuss.elastic.co/t/logstash-help-with-geoip-mapping-from-csv-file/169229 "2019-02-21T15:06:42Z")
Hi, sorry for asking, but went through code, help, FAQ and i still cant make it work. I have CSV file with two fields which contains IP address. When importing from Kibana directly i can make this fields to be recognized… --- ## [\_grokparsefailure tag](https://discuss.elastic.co/t/grokparsefailure-tag/168010)
**Author:** [@Nat](https://discuss.elastic.co/u/Nat)\ **Replies:** 5\ **Last updated:** [February 21, 2019, 3:02pm UTC](https://discuss.elastic.co/t/grokparsefailure-tag/168010 "2019-02-21T15:02:28Z")
Hello, I am trying to parse the message from some logs and split the message in different fields. I am using some custom made grok patterns that should apply to all the messages (tested at grok debugger). Nevertheless … --- ## [How to pass multiple csv files with different names and different content?](https://discuss.elastic.co/t/how-to-pass-multiple-csv-files-with-different-names-and-different-content/169389)
**Author:** [@Thodoris\_Alexopoulos](https://discuss.elastic.co/u/Thodoris_Alexopoulos)\ **Replies:** 5\ **Last updated:** [February 21, 2019, 2:42pm UTC](https://discuss.elastic.co/t/how-to-pass-multiple-csv-files-with-different-names-and-different-content/169389 "2019-02-21T14:42:35Z")
Hello, I want to pass into Elastisearch several CSV files with a different format each other. Till now, I have created a Logstash conf file that handles only one CSV with a specific name under a specific path. But when … --- ## [Duration calculation failing](https://discuss.elastic.co/t/duration-calculation-failing/169411)
**Author:** [@Rudolf\_Reddy\_Macejka](https://discuss.elastic.co/u/Rudolf_Reddy_Macejka)\ **Replies:** 1\ **Last updated:** [February 21, 2019, 2:09pm UTC](https://discuss.elastic.co/t/duration-calculation-failing/169411 "2019-02-21T14:09:19Z")
Hello, I am trying to calculate duration by grog and inspirated by another topis here. So I have this code: date { match =\> \[ "time","UNIX\_MS" \] target =\> "@timestamp" remove\_field =\> \[ "time" \] } m… --- ## [Unable to set sub-condition in condition](https://discuss.elastic.co/t/unable-to-set-sub-condition-in-condition/169400)
**Author:** [@Rudolf\_Reddy\_Macejka](https://discuss.elastic.co/u/Rudolf_Reddy_Macejka)\ **Replies:** 1\ **Last updated:** [February 21, 2019, 2:07pm UTC](https://discuss.elastic.co/t/unable-to-set-sub-condition-in-condition/169400 "2019-02-21T14:07:20Z")
Hello folks, I am fighting with one problem in logstash. I would like to set a sub-condition in existing condition. if \[type\] == "system" { mutate { rename =\> { "instCpu" =\> "cpuTotal" } rename =\> { … --- ## [Grok Type Conversion Not Working (logstash-6.6.0)](https://discuss.elastic.co/t/grok-type-conversion-not-working-logstash-6-6-0/169330)
**Author:** [@Hitesh\_Khatri](https://discuss.elastic.co/u/Hitesh_Khatri)\ **Replies:** 1\ **Last updated:** [February 21, 2019, 1:39pm UTC](https://discuss.elastic.co/t/grok-type-conversion-not-working-logstash-6-6-0/169330 "2019-02-21T13:39:33Z")
I have defined type conversion for id field to long in my grok pattern, still result is showing string. What am I doing wrong here? Please help. Input: 127.0.0.1 \[http-bio-8080-exec-33091\] INFO 3747742999301509130 … --- ## [Date parsing failure error](https://discuss.elastic.co/t/date-parsing-failure-error/169203)
**Author:** [@vsingh](https://discuss.elastic.co/u/vsingh)\ **Replies:** 5\ **Last updated:** [February 21, 2019, 1:16pm UTC](https://discuss.elastic.co/t/date-parsing-failure-error/169203 "2019-02-21T13:16:35Z")
Hello there, I am getting dateparsingfailure error when running logstash-6.4.0. This is my database column and its values sno reference\_id dateofbirth post\_date 7979 2671254433 1990-01-31 2017-01-19 10… --- ## [Logstash start error!](https://discuss.elastic.co/t/logstash-start-error/168973)
**Author:** [@9129b75705fb87ec814a](https://discuss.elastic.co/u/9129b75705fb87ec814a)\ **Replies:** 3\ **Last updated:** [February 21, 2019, 1:12pm UTC](https://discuss.elastic.co/t/logstash-start-error/168973 "2019-02-21T13:12:22Z")
i try to start logstash with config file but it can't work ERROR: \[2019-02-19T16:59:48,269\]\[ERROR\]\[logstash.agent \] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :excep… --- ## [Logstash date filter conversion](https://discuss.elastic.co/t/logstash-date-filter-conversion/169402)
**Author:** [@vasudevan](https://discuss.elastic.co/u/vasudevan)\ **Replies:** 0\ **Last updated:** [February 21, 2019, 11:39am UTC](https://discuss.elastic.co/t/logstash-date-filter-conversion/169402 "2019-02-21T11:39:22Z")
We have log file in which we have to capture the first line matching "TIMESTAMP\_ISO8601" against build\_StartTime filed and last line matching "TIMESTAMP\_ISO8601" against build\_EndTime filed. After this we have to calcula… --- ## [Permission denied - /etc/logstash/conf.d/logstash.conf](https://discuss.elastic.co/t/permission-denied-etc-logstash-conf-d-logstash-conf/169392)
**Author:** [@shaonbean](https://discuss.elastic.co/u/shaonbean)\ **Replies:** 1\ **Last updated:** [February 21, 2019, 10:50am UTC](https://discuss.elastic.co/t/permission-denied-etc-logstash-conf-d-logstash-conf/169392 "2019-02-21T10:50:36Z")
\[2019-02-21T18:44:36,180\]\[INFO \]\[logstash.runner \] Starting Logstash {"logstash.version"=\>"6.6.1"} \[2019-02-21T18:44:37,623\]\[ERROR\]\[logstash.config.sourceloader\] Could not fetch all the sources {:exception=\>Err… --- ## [Logstash date filter not working](https://discuss.elastic.co/t/logstash-date-filter-not-working/169381)
**Author:** [@vasudevan](https://discuss.elastic.co/u/vasudevan)\ **Replies:** 0\ **Last updated:** [February 21, 2019, 10:24am UTC](https://discuss.elastic.co/t/logstash-date-filter-not-working/169381 "2019-02-21T10:24:18Z")
We have log file in which we have to capture the first line matching "TIMESTAMP\_ISO8601" against build\_StartTime filed and last line matching "TIMESTAMP\_ISO8601" against build\_EndTime filed. After this we have to calcula… --- ## [Need to send index to 2 ES from Logstash](https://discuss.elastic.co/t/need-to-send-index-to-2-es-from-logstash/169368)
**Author:** [@Loudone](https://discuss.elastic.co/u/Loudone)\ **Replies:** 4\ **Last updated:** [February 21, 2019, 10:23am UTC](https://discuss.elastic.co/t/need-to-send-index-to-2-es-from-logstash/169368 "2019-02-21T10:23:33Z")
Hello All, I'm currently trying to send index from logstash to 2 Elasticsearch cluster like below: the problem is, in output logstash's section, it's possible to set in Hosts two ES cluster, but in loadbalance mode. … --- ## [Getting issue while creating the index for jsonb column from postgres database](https://discuss.elastic.co/t/getting-issue-while-creating-the-index-for-jsonb-column-from-postgres-database/169380)
**Author:** [@MacJio](https://discuss.elastic.co/u/MacJio)\ **Replies:** 0\ **Last updated:** [February 21, 2019, 10:21am UTC](https://discuss.elastic.co/t/getting-issue-while-creating-the-index-for-jsonb-column-from-postgres-database/169380 "2019-02-21T10:21:10Z")
In my database table one column type is jsonb . I tried to index the same table by using logstash. I am getting below error . Please help on this. Error : Could not index event to Elasticsearch. "\_id"=\>"d999b3f2-e903… --- ## [How to stop fields from kv filter splitting](https://discuss.elastic.co/t/how-to-stop-fields-from-kv-filter-splitting/169197)
**Author:** [@priya08](https://discuss.elastic.co/u/priya08)\ **Replies:** 2\ **Last updated:** [February 21, 2019, 10:12am UTC](https://discuss.elastic.co/t/how-to-stop-fields-from-kv-filter-splitting/169197 "2019-02-21T10:12:56Z")
Hi Team, I am using KV filter for my log statement , sample log statement \[INFO\]|logtype:SYSLOG|datetime:2019-02-20T16:59:55.045+0530|hostname:ABCD|threadId:87-pool-3-thread-8(main)|userId:null|payload:\ **Author:** [@vasudevan](https://discuss.elastic.co/u/vasudevan)\ **Replies:** 0\ **Last updated:** [February 21, 2019, 9:34am UTC](https://discuss.elastic.co/t/converting-string-to-date-in-logstash/169357 "2019-02-21T09:34:32Z") We have log file in which we have to capture the first line matching "TIMESTAMP\_ISO8601" against build\_StartTime filed and last line matching "TIMESTAMP\_ISO8601" against build\_EndTime filed. After this we have to calcula… --- ## [Pushing logs stored within Minio to be viewed in Kibana](https://discuss.elastic.co/t/pushing-logs-stored-within-minio-to-be-viewed-in-kibana/169354)
**Author:** [@Hanrahank](https://discuss.elastic.co/u/Hanrahank)\ **Replies:** 0\ **Last updated:** [February 21, 2019, 9:23am UTC](https://discuss.elastic.co/t/pushing-logs-stored-within-minio-to-be-viewed-in-kibana/169354 "2019-02-21T09:23:12Z")
I've been trying to find ways of pushing logs from Minio through ELK Stack. I've been looking at making Minio a possibly custom S3 input in a Logstash Pipeline. However, with no avail. Is this possible to be done within … --- ## [Logstash change data in jdbc plugin](https://discuss.elastic.co/t/logstash-change-data-in-jdbc-plugin/169352)
**Author:** [@sahere37](https://discuss.elastic.co/u/sahere37)\ **Replies:** 0\ **Last updated:** [February 21, 2019, 9:02am UTC](https://discuss.elastic.co/t/logstash-change-data-in-jdbc-plugin/169352 "2019-02-21T09:02:58Z")
hi all i am using logstash jdbc plugin to ship data from sql server database to elasticsearch using following: input { jdbc { jdbc\_driver\_library =\> "F:\\driver\\sqljdbc\_6.0\\enu\\jre8\\sqljdbc42.jar" jdbc\_driver\_c… --- ## [ERROR while importing smtp logs](https://discuss.elastic.co/t/error-while-importing-smtp-logs/169340)
**Author:** [@Vikash\_Singh1](https://discuss.elastic.co/u/Vikash_Singh1)\ **Replies:** 0\ **Last updated:** [February 21, 2019, 7:03am UTC](https://discuss.elastic.co/t/error-while-importing-smtp-logs/169340 "2019-02-21T07:03:25Z")
DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception..What does it actually means?? --- ## [A lot of empty messages with AWS SQS](https://discuss.elastic.co/t/a-lot-of-empty-messages-with-aws-sqs/169157)
**Author:** [@ep4sh](https://discuss.elastic.co/u/ep4sh)\ **Replies:** 1\ **Last updated:** [February 21, 2019, 6:23am UTC](https://discuss.elastic.co/t/a-lot-of-empty-messages-with-aws-sqs/169157 "2019-02-21T06:23:02Z")
I have a lot of empty messages on my Logstash indexers, I am using AWS SQS input and my attribute on SQS tier (i think it is enough time to process) ReceiveMessageWaitTimeSeconds == 10 sec. Is it a responsibility o… --- ## [How to run Incremental update(addition of new rows) + updation of existing rows simultaneously, when running SQL query through logstash jdbc plug in](https://discuss.elastic.co/t/how-to-run-incremental-update-addition-of-new-rows-updation-of-existing-rows-simultaneously-when-running-sql-query-through-logstash-jdbc-plug-in/168985)
**Author:** [@ApekshaB](https://discuss.elastic.co/u/ApekshaB)\ **Replies:** 2\ **Last updated:** [February 21, 2019, 5:58am UTC](https://discuss.elastic.co/t/how-to-run-incremental-update-addition-of-new-rows-updation-of-existing-rows-simultaneously-when-running-sql-query-through-logstash-jdbc-plug-in/168985 "2019-02-21T05:58:16Z")
Dear Team, I have a table in oracle DB, which consists of 4 columns, say-- 'Serial\_no', 'Transaction\_Amt', 'User\_Name', 'modification\_timestamp'. Now i run sql select query in logstash conf file(using jdbc connection) … --- ## [Logstash shutdown as soon as input file is place into input directory](https://discuss.elastic.co/t/logstash-shutdown-as-soon-as-input-file-is-place-into-input-directory/169053)
**Author:** [@KevinT1](https://discuss.elastic.co/u/KevinT1)\ **Replies:** 2\ **Last updated:** [February 20, 2019, 2:31pm UTC](https://discuss.elastic.co/t/logstash-shutdown-as-soon-as-input-file-is-place-into-input-directory/169053 "2019-02-20T14:31:29Z")
Hi, Need help with file input of logstash ver 2.2.1. Steps to simulate it Empty the input directory Start logstash (in debug where log messages are listed) Place input file into input directory. logstash is shutdown… --- ## [Netflow Installation on Windows Logstash](https://discuss.elastic.co/t/netflow-installation-on-windows-logstash/169307)
**Author:** [@Jordan\_Reich](https://discuss.elastic.co/u/Jordan_Reich)\ **Replies:** 0\ **Last updated:** [February 21, 2019, 12:04am UTC](https://discuss.elastic.co/t/netflow-installation-on-windows-logstash/169307 "2019-02-21T00:04:14Z")
I have found documentation but I don't appear to be able to find anything relevant to setting up the netflow installation on a windows machine. Can someone point me in the right direction? Thanks! --- ## [Display pr-BR Content in the strings in Kibana](https://discuss.elastic.co/t/display-pr-br-content-in-the-strings-in-kibana/168834)
**Author:** [@ssasporta](https://discuss.elastic.co/u/ssasporta)\ **Replies:** 4\ **Last updated:** [February 20, 2019, 10:24pm UTC](https://discuss.elastic.co/t/display-pr-br-content-in-the-strings-in-kibana/168834 "2019-02-20T22:24:28Z")
Some of my input strings are written in Brazilian Portuguese. The punctuation characters are displayed as gibberish. For example: ORGANIZACAOGRUPOCOORDENADOR Sustenta��o The value should be: Sustentação How c… --- ## [Get timestamp from "embedded" syslog entry](https://discuss.elastic.co/t/get-timestamp-from-embedded-syslog-entry/169194)
**Author:** [@Hyllegaard](https://discuss.elastic.co/u/Hyllegaard)\ **Replies:** 6\ **Last updated:** [February 20, 2019, 10:01pm UTC](https://discuss.elastic.co/t/get-timestamp-from-embedded-syslog-entry/169194 "2019-02-20T22:01:15Z")
Hi. We have just started using the ELK stack, so please excuse the noobiness of these questions :slight\_smile: and we have a situation where we have to use syslog to forward the logs from a SOLR server. The format of … --- ## [Correct way of Logstash persisted queue performance testing](https://discuss.elastic.co/t/correct-way-of-logstash-persisted-queue-performance-testing/169113)
**Author:** [@alexpanas](https://discuss.elastic.co/u/alexpanas)\ **Replies:** 5\ **Last updated:** [February 20, 2019, 9:16pm UTC](https://discuss.elastic.co/t/correct-way-of-logstash-persisted-queue-performance-testing/169113 "2019-02-20T21:16:26Z")
Hello, I am trying to measure Logstash 6.6 performance to find the performance impact of enabling persisted queues on Logstash 6.6. To eliminate any influence of elasticsearch environment - all tests are being output… --- ## [Newbie Issue with multiple dissects](https://discuss.elastic.co/t/newbie-issue-with-multiple-dissects/169245)
**Author:** [@jerry.browne](https://discuss.elastic.co/u/jerry.browne)\ **Replies:** 2\ **Last updated:** [February 20, 2019, 7:30pm UTC](https://discuss.elastic.co/t/newbie-issue-with-multiple-dissects/169245 "2019-02-20T19:30:59Z")
I am brand new to logstash so apologies in advance for what I'm sure will turn out to be "dumb" questions. I have a very simple log file with some | separated data in it. Like this: 1550613490|MANAGER|1|Local/901@inte… --- ## [Conditionals in filter clause](https://discuss.elastic.co/t/conditionals-in-filter-clause/169238)
**Author:** [@humartinez](https://discuss.elastic.co/u/humartinez)\ **Replies:** 4\ **Last updated:** [February 20, 2019, 5:37pm UTC](https://discuss.elastic.co/t/conditionals-in-filter-clause/169238 "2019-02-20T17:37:25Z")
Hi Im trying to do some conditional processing in the filter section in logstash. The logic is if some field exist do something else other. if \[extended\_tweet.full\_text\] { nlp { source =\> "extended\_tweet.full\_t… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=595) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=597)