# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=325 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 326 --- ## [Handling exception from HTTP logstash 6.8](https://discuss.elastic.co/t/handling-exception-from-http-logstash-6-8/242259)
**Author:** [@pvaithee07](https://discuss.elastic.co/u/pvaithee07)\ **Replies:** 4\ **Last updated:** [July 22, 2020, 11:18pm UTC](https://discuss.elastic.co/t/handling-exception-from-http-logstash-6-8/242259 "2020-07-22T23:18:17Z")
Hello - I am trying to handle exception in my logstash after HTTP filter plugin. I am using Logstash version 6.8. I have couple of question on this - How to handle exception if the HTTP response code is other than 200… --- ## [Not able to get the CSV values in the filter columns](https://discuss.elastic.co/t/not-able-to-get-the-csv-values-in-the-filter-columns/242263)
**Author:** [@Harshil\_Narielwala](https://discuss.elastic.co/u/Harshil_Narielwala)\ **Replies:** 1\ **Last updated:** [July 22, 2020, 10:40pm UTC](https://discuss.elastic.co/t/not-able-to-get-the-csv-values-in-the-filter-columns/242263 "2020-07-22T22:40:26Z")
My logstash configuration is as below : input { beats { port =\> "5044" } } filter { csv { columns =\> \["REQUEST\_ID","SERVICE\_NAME","OPERATION\_NAME","OPERATION\_TYPE","REQUEST\_PAYLOAD","LOGGED\_BY","LOGGED\_ON"\] } } … --- ## [Logstash config: Elastic search indices not being properly populated from multiple http\_poller inputs](https://discuss.elastic.co/t/logstash-config-elastic-search-indices-not-being-properly-populated-from-multiple-http-poller-inputs/241929)
**Author:** [@AnEpicK9](https://discuss.elastic.co/u/AnEpicK9)\ **Replies:** 8\ **Last updated:** [July 22, 2020, 7:25pm UTC](https://discuss.elastic.co/t/logstash-config-elastic-search-indices-not-being-properly-populated-from-multiple-http-poller-inputs/241929 "2020-07-22T19:25:07Z")
Currently, I have two http\_poller input blocks with a url for a live and backup instance of our app in my logstash config. What I'm then doing is I'm trying to output to two different elastic search indices: logstash\_pol… --- ## [Auto\_flush\_interval is invalid with Logstash 7.5.1](https://discuss.elastic.co/t/auto-flush-interval-is-invalid-with-logstash-7-5-1/242057)
**Author:** [@thanhlam](https://discuss.elastic.co/u/thanhlam)\ **Replies:** 16\ **Last updated:** [July 22, 2020, 7:09pm UTC](https://discuss.elastic.co/t/auto-flush-interval-is-invalid-with-logstash-7-5-1/242057 "2020-07-22T19:09:53Z")
Hello, I've been running Logstash 7.5.1 on IBM Power systems for a couple of months. But, the last time I restarted Logstash, it kept shutdown and restarting. So I run it with config.debug = true log.level: trace and … --- ## [Pipeline not discovering with correct paths](https://discuss.elastic.co/t/pipeline-not-discovering-with-correct-paths/242243)
**Author:** [@mattr](https://discuss.elastic.co/u/mattr)\ **Replies:** 1\ **Last updated:** [July 22, 2020, 6:26pm UTC](https://discuss.elastic.co/t/pipeline-not-discovering-with-correct-paths/242243 "2020-07-22T18:26:35Z")
I am having trouble with the logstash pipeline. I've had the pipeline working with the exact same settings on a different server. The config works when I run it manually using the command line command. The discoverer d… --- ## [Timestamp conversion](https://discuss.elastic.co/t/timestamp-conversion/242136)
**Author:** [@ishank](https://discuss.elastic.co/u/ishank)\ **Replies:** 1\ **Last updated:** [July 22, 2020, 5:37pm UTC](https://discuss.elastic.co/t/timestamp-conversion/242136 "2020-07-22T17:37:57Z")
Hi The data in my csv file is having the timestamp in format: 2019-04-11 12:00 AM (There are 3 spaces between the date & time) My logstash configuration for same: filter{ date { match =\> \["Time", "yyyy-MM-… --- ## [How to create exact replica of data table from mysql mariaDB](https://discuss.elastic.co/t/how-to-create-exact-replica-of-data-table-from-mysql-mariadb/241903)
**Author:** [@Sahilvaishnav](https://discuss.elastic.co/u/Sahilvaishnav)\ **Replies:** 1\ **Last updated:** [July 22, 2020, 1:56pm UTC](https://discuss.elastic.co/t/how-to-create-exact-replica-of-data-table-from-mysql-mariadb/241903 "2020-07-22T13:56:21Z")
Hello, Scenario : I have a database in mysql below is the structure : JOBID Wokflow Ingest Formula Load modification\_time insertion\_time 1 Pipeline1 Completed Completed Completed 2020-07-18 17:12:35 2020-07-18… --- ## [Grokparsefailure and type problems in logstash configuration file](https://discuss.elastic.co/t/grokparsefailure-and-type-problems-in-logstash-configuration-file/242200)
**Author:** [@Mathis\_Gourc](https://discuss.elastic.co/u/Mathis_Gourc)\ **Replies:** 0\ **Last updated:** [July 22, 2020, 1:18pm UTC](https://discuss.elastic.co/t/grokparsefailure-and-type-problems-in-logstash-configuration-file/242200 "2020-07-22T13:18:27Z")
Hi everybody, I have several problems with my configuration file. My goal is to parse three types of logs (for the moment). Here they are : \[29/05/2020 07:41:51.354\] - ih912865 - 10.107.119.121 - 93 - Transaction 7635 … --- ## [Logstash AWS Elastic Search issue](https://discuss.elastic.co/t/logstash-aws-elastic-search-issue/242164)
**Author:** [@ggiff](https://discuss.elastic.co/u/ggiff)\ **Replies:** 1\ **Last updated:** [July 22, 2020, 11:17am UTC](https://discuss.elastic.co/t/logstash-aws-elastic-search-issue/242164 "2020-07-22T11:17:59Z")
I have set up Logstash to send user inputs to AWS ElasticSearch, but getting the following issue: \[WARN \] 2020-07-22 09:27:50.704 \[Ruby-0-Thread-4: :1\] elasticsearch - Attempted to resurrect connection to dead ES instan… --- ## [Drop message content](https://discuss.elastic.co/t/drop-message-content/242158)
**Author:** [@Champion\_Xie](https://discuss.elastic.co/u/Champion_Xie)\ **Replies:** 1\ **Last updated:** [July 22, 2020, 9:35am UTC](https://discuss.elastic.co/t/drop-message-content/242158 "2020-07-22T09:35:22Z")
hi all, message content 07/20/20 23:57:48 INFO transfercity:sh(city/cityTransferQueryBusiness.cpp:72) how to only get 07/20/20 23:57:48 INFO transfercity:sh --- ## [File output with data from field](https://discuss.elastic.co/t/file-output-with-data-from-field/241895)
**Author:** [@elk2](https://discuss.elastic.co/u/elk2)\ **Replies:** 2\ **Last updated:** [July 21, 2020, 7:47am UTC](https://discuss.elastic.co/t/file-output-with-data-from-field/241895 "2020-07-21T07:47:47Z")
I collect different logs from various sources. I replaced logstash timestamp with machine logs timestamp to have time based events into elasticsearch and I added a "processed\_at" field with a part of logstash timestamp. … --- ## [Log messages with same index timestamps are not in the order as in the actual log file](https://discuss.elastic.co/t/log-messages-with-same-index-timestamps-are-not-in-the-order-as-in-the-actual-log-file/242000)
**Author:** [@Aj\_t](https://discuss.elastic.co/u/Aj_t)\ **Replies:** 9\ **Last updated:** [July 22, 2020, 9:12am UTC](https://discuss.elastic.co/t/log-messages-with-same-index-timestamps-are-not-in-the-order-as-in-the-actual-log-file/242000 "2020-07-22T09:12:27Z")
I have a filebeat \> logstash \> elastic set up I am using to process multi-line logs. I can see the logs in kibana, but they are not in the right order as the actual log. Here's my filebeat.yml file: filebeat.inputs: - … --- ## [How to get string from path](https://discuss.elastic.co/t/how-to-get-string-from-path/242156)
**Author:** [@Champion\_Xie](https://discuss.elastic.co/u/Champion_Xie)\ **Replies:** 0\ **Last updated:** [July 22, 2020, 8:59am UTC](https://discuss.elastic.co/t/how-to-get-string-from-path/242156 "2020-07-22T08:59:50Z")
hi all, My application log file name format is room1\_num\_perf.log, i want get the string 'room1' ,and then the string as a new field into ES --- ## [GeoIP Filter Mapping Error. What dynamic mapping is required?](https://discuss.elastic.co/t/geoip-filter-mapping-error-what-dynamic-mapping-is-required/242144)
**Author:** [@clearedskies](https://discuss.elastic.co/u/clearedskies)\ **Replies:** 2\ **Last updated:** [July 22, 2020, 7:47am UTC](https://discuss.elastic.co/t/geoip-filter-mapping-error-what-dynamic-mapping-is-required/242144 "2020-07-22T07:47:46Z")
I have been attempting to parse some WAF logs so they fit with the ECS Schema which I have finally managed to do. However, my final issue is to do with the GeoIP Filter. I wish to utilise the Maps element of Kibana Visua… --- ## [How to parse nested logs with JSON through logstash](https://discuss.elastic.co/t/how-to-parse-nested-logs-with-json-through-logstash/241781)
**Author:** [@Ranjan\_Bansal](https://discuss.elastic.co/u/Ranjan_Bansal)\ **Replies:** 4\ **Last updated:** [July 22, 2020, 4:59am UTC](https://discuss.elastic.co/t/how-to-parse-nested-logs-with-json-through-logstash/241781 "2020-07-22T04:59:04Z")
Hello All, I am facing issue while parsing nested JSON in my logs. I checked some of the posts over internet regarding nested JSON but none actually resolved my problem. There might be the case I am making some mistake … --- ## [Exporting all documents from index but Excel is limited](https://discuss.elastic.co/t/exporting-all-documents-from-index-but-excel-is-limited/241855)
**Author:** [@ion\_popa](https://discuss.elastic.co/u/ion_popa)\ **Replies:** 1\ **Last updated:** [July 22, 2020, 4:17am UTC](https://discuss.elastic.co/t/exporting-all-documents-from-index-but-excel-is-limited/241855 "2020-07-22T04:17:57Z")
Hello, I've been assigned a task to export all the documents from an index to a .csv file so I'm using Logstash for that but when I open up the .csv using Excel, I've reached the limit of 1,048,576 rows per sheet and I … --- ## [Parsing Error when changing Julian time on Logstashfilter](https://discuss.elastic.co/t/parsing-error-when-changing-julian-time-on-logstashfilter/242112)
**Author:** [@ben\_men](https://discuss.elastic.co/u/ben_men)\ **Replies:** 0\ **Last updated:** [July 21, 2020, 11:43pm UTC](https://discuss.elastic.co/t/parsing-error-when-changing-julian-time-on-logstashfilter/242112 "2020-07-21T23:43:36Z")
I was trying to parse Julian time in Logstash and got dateparsefailure. Here is the log event I have 2020/197/21:13:57.605 FINE jdk.internal.event.EventHelper.logX509CertificateEvent Here is the filter I have: a… --- ## [Logstash - parse to fields with grok](https://discuss.elastic.co/t/logstash-parse-to-fields-with-grok/242033)
**Author:** [@elk6](https://discuss.elastic.co/u/elk6)\ **Replies:** 5\ **Last updated:** [July 21, 2020, 8:32pm UTC](https://discuss.elastic.co/t/logstash-parse-to-fields-with-grok/242033 "2020-07-21T20:32:24Z")
TL;DR at bottom I have a custom generated log file that I made that lists commands users run (along with some other thing). For that, I have got a grok script that successfully parses the message into fields. I've start… --- ## [Max event size for tcp input plugin](https://discuss.elastic.co/t/max-event-size-for-tcp-input-plugin/241773)
**Author:** [@Jehutywong](https://discuss.elastic.co/u/Jehutywong)\ **Replies:** 1\ **Last updated:** [July 21, 2020, 3:52pm UTC](https://discuss.elastic.co/t/max-event-size-for-tcp-input-plugin/241773 "2020-07-21T15:52:40Z")
Logstash version 7.6 I am sending a big json over TCP to Logstash. Somehow only partial of the event was able to output. roughly 60KB. The rest of the event was dicarded. Any way i can increase this limit. I tried to i… --- ## [From logstash i want forward message to an application that read syslog in from port 514 ... the applictaion read and support Octet counting or LF ... how to setup logstah to send Octect or LF?](https://discuss.elastic.co/t/from-logstash-i-want-forward-message-to-an-application-that-read-syslog-in-from-port-514-the-applictaion-read-and-support-octet-counting-or-lf-how-to-setup-logstah-to-send-octect-or-lf/242063)
**Author:** [@Maurizio\_Tarducci](https://discuss.elastic.co/u/Maurizio_Tarducci)\ **Replies:** 0\ **Last updated:** [July 21, 2020, 3:51pm UTC](https://discuss.elastic.co/t/from-logstash-i-want-forward-message-to-an-application-that-read-syslog-in-from-port-514-the-applictaion-read-and-support-octet-counting-or-lf-how-to-setup-logstah-to-send-octect-or-lf/242063 "2020-07-21T15:51:18Z")
from logstash i want forward message to an application that read syslog in from port 514 ... the applictaion read and support Octet counting or LF ... how to setup logstah to send Octect or LF ? --- ## [Gelf output to Logstash not working](https://discuss.elastic.co/t/gelf-output-to-logstash-not-working/242061)
**Author:** [@pointonj](https://discuss.elastic.co/u/pointonj)\ **Replies:** 0\ **Last updated:** [July 21, 2020, 3:44pm UTC](https://discuss.elastic.co/t/gelf-output-to-logstash-not-working/242061 "2020-07-21T15:44:58Z")
Hi, I'm trying to get the logs from my Graylog server into 3 Logstash Servers that are load balanced. I'm not sure if this is an issue with Graylog or Logstash but as i've posted this on the Graylog community and got no… --- ## [Salesforce and Logstash integration](https://discuss.elastic.co/t/salesforce-and-logstash-integration/242055)
**Author:** [@vtrezza](https://discuss.elastic.co/u/vtrezza)\ **Replies:** 0\ **Last updated:** [July 21, 2020, 3:22pm UTC](https://discuss.elastic.co/t/salesforce-and-logstash-integration/242055 "2020-07-21T15:22:23Z")
Hi, I have to integrate Salesforce and Logstash in order to sync data from Salesforce to Logstash. The only possible way I have found is using the plugin "Salesforce input plugin". This plugin is working on making a S… --- ## [Handling IBM MQ messages with MQRFH2 header in logstash](https://discuss.elastic.co/t/handling-ibm-mq-messages-with-mqrfh2-header-in-logstash/242028)
**Author:** [@rohitarorait82](https://discuss.elastic.co/u/rohitarorait82)\ **Replies:** 1\ **Last updated:** [July 21, 2020, 1:33pm UTC](https://discuss.elastic.co/t/handling-ibm-mq-messages-with-mqrfh2-header-in-logstash/242028 "2020-07-21T13:33:31Z")
Hi All, Can you please suggest , how to handle incoming IBM MQ messages MQRFH2 header in logstash, It is working fine when I pick a normal message without MQRFH2 header but whenever client is sending MQRFH2 header, thi… --- ## [HTTP Proxy twitter client error for logstash-input-twitter plugin](https://discuss.elastic.co/t/http-proxy-twitter-client-error-for-logstash-input-twitter-plugin/242027)
**Author:** [@shaswatgupta](https://discuss.elastic.co/u/shaswatgupta)\ **Replies:** 0\ **Last updated:** [July 21, 2020, 12:58pm UTC](https://discuss.elastic.co/t/http-proxy-twitter-client-error-for-logstash-input-twitter-plugin/242027 "2020-07-21T12:58:46Z")
Logstash log: \[2020-07-21T12:08:19,838\]\[WARN \]\[logstash.inputs.twitter \] Twitter client error {:message=\>"End of file reached", :exception=\>"EOFError", :backtrace=\>\["org/jruby/RubyIO.java:2889:in \`readpartial'", "/us… --- ## [Error while starting Logstash in WIndows](https://discuss.elastic.co/t/error-while-starting-logstash-in-windows/242006)
**Author:** [@Mitan\_Kumar\_Batu](https://discuss.elastic.co/u/Mitan_Kumar_Batu)\ **Replies:** 0\ **Last updated:** [July 21, 2020, 10:40am UTC](https://discuss.elastic.co/t/error-while-starting-logstash-in-windows/242006 "2020-07-21T10:40:42Z")
Getting below error while starting Logstash:- \[ERROR\] 2020-07-21 16:04:29.491 \[main\] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (GemspecError) \[!\] There was an error wh… --- ## [No config files found and no configuration source found](https://discuss.elastic.co/t/no-config-files-found-and-no-configuration-source-found/242003)
**Author:** [@Bigranawab](https://discuss.elastic.co/u/Bigranawab)\ **Replies:** 0\ **Last updated:** [July 21, 2020, 10:27am UTC](https://discuss.elastic.co/t/no-config-files-found-and-no-configuration-source-found/242003 "2020-07-21T10:27:15Z")
\ WARNING: Could not find logstash.yml which is typically located in $LS\_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration… --- ## [Logstash consumes too much CPU while Redis is down](https://discuss.elastic.co/t/logstash-consumes-too-much-cpu-while-redis-is-down/241992)
**Author:** [@gabor-T-mandity](https://discuss.elastic.co/u/gabor-T-mandity)\ **Replies:** 0\ **Last updated:** [July 21, 2020, 9:02am UTC](https://discuss.elastic.co/t/logstash-consumes-too-much-cpu-while-redis-is-down/241992 "2020-07-21T09:02:34Z")
We have multple pipelines, which has the same structure: Redis cache in seperate environments and one centralized ELK stack. To save up some money, we shut down our dev environments every day. Our problem is, that while… --- ## [Logstash encountered retryable error](https://discuss.elastic.co/t/logstash-encountered-retryable-error/241991)
**Author:** [@Gauti](https://discuss.elastic.co/u/Gauti)\ **Replies:** 0\ **Last updated:** [July 21, 2020, 9:02am UTC](https://discuss.elastic.co/t/logstash-encountered-retryable-error/241991 "2020-07-21T09:02:08Z")
hi All, We are ingesting documents using logstash, all of a sudden i'm getting the below error. \[\[main\]\>worker1\] elasticsearch - Encountered a retryable error. Will Retry with exponential backoff {:code=\>400, :url=\>"h… --- ## [”\_grokparsefailure” even though the grok pattern works](https://discuss.elastic.co/t/grokparsefailure-even-though-the-grok-pattern-works/241647)
**Author:** [@Mathis\_Gourc](https://discuss.elastic.co/u/Mathis_Gourc)\ **Replies:** 4\ **Last updated:** [July 21, 2020, 8:30am UTC](https://discuss.elastic.co/t/grokparsefailure-even-though-the-grok-pattern-works/241647 "2020-07-21T08:30:35Z")
I am trying to parse different logs lines from two different type of file : slave and master. I did test my pattern in the Grok Dubugger and it is working fine but tags field in kibana is \_grokparsefailure. Here is my c… --- ## [Csv filter don't work in realtime](https://discuss.elastic.co/t/csv-filter-dont-work-in-realtime/241979)
**Author:** [@elk2](https://discuss.elastic.co/u/elk2)\ **Replies:** 0\ **Last updated:** [July 21, 2020, 7:43am UTC](https://discuss.elastic.co/t/csv-filter-dont-work-in-realtime/241979 "2020-07-21T07:43:48Z")
I send a csv file with filebeat to logstash but csv filter plugin fails when it tries to parse csv with error "unclosed quoted field on line 1" If I save csv in file with logstash and with another conf I use csv filter … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=324) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=326)