# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=720 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 721 --- ## [Data parse error \["\_dateparsefailure"\]](https://discuss.elastic.co/t/data-parse-error-dateparsefailure/139924)
**Author:** [@Gokul\_15](https://discuss.elastic.co/u/Gokul_15)\ **Replies:** 8\ **Last updated:** [July 18, 2018, 4:40am UTC](https://discuss.elastic.co/t/data-parse-error-dateparsefailure/139924 "2018-07-18T04:40:32Z")
data parse error \["\_dateparsefailure"\] filter{ date{ match=\>\["date","yyyy-MM-dd HH:mm:ss:SSS Z"\] target=\>"logdata" } mutate { remove\_field =\> \[ "date" \] } where date inside the match is the field in db.. "date":"2… --- ## [Logstash log output is truncated](https://discuss.elastic.co/t/logstash-log-output-is-truncated/140431)
**Author:** [@aliaramli](https://discuss.elastic.co/u/aliaramli)\ **Replies:** 0\ **Last updated:** [July 18, 2018, 3:51am UTC](https://discuss.elastic.co/t/logstash-log-output-is-truncated/140431 "2018-07-18T03:51:39Z")
Hi, How can i set the logstash output log to be in full message, i always see "\[truncated 7670 bytes\]" and its a bit hard for me to troubleshoot input data parse issue. Thank You! --- ## [Logstash input gelf](https://discuss.elastic.co/t/logstash-input-gelf/140430)
**Author:** [@springCkWyy](https://discuss.elastic.co/u/springCkWyy)\ **Replies:** 0\ **Last updated:** [July 18, 2018, 3:46am UTC](https://discuss.elastic.co/t/logstash-input-gelf/140430 "2018-07-18T03:46:14Z")
Now I try use gelf plugin to receicve Data。 I means should GELF-ormat Message。 And Now I build a spring boot Project and the log as the input for logstash. But I dont konw what the " GELF-ormat Message" like? And Can … --- ## [GeoIP issue on logstash conf file](https://discuss.elastic.co/t/geoip-issue-on-logstash-conf-file/140031)
**Author:** [@Abdur\_Raqeeb1](https://discuss.elastic.co/u/Abdur_Raqeeb1)\ **Replies:** 11\ **Last updated:** [July 18, 2018, 3:38am UTC](https://discuss.elastic.co/t/geoip-issue-on-logstash-conf-file/140031 "2018-07-18T03:38:19Z")
Hi, Myself getting an error after giving an geoIP database separately for geoip city and geoIP country. Please find the error message below /usr/share/logstash/bin/logstash -t -f /etc/logstash/conf.d/beats.conf WARNI… --- ## [Filtered and unfiltered outputs](https://discuss.elastic.co/t/filtered-and-unfiltered-outputs/140239)
**Author:** [@rocknroll](https://discuss.elastic.co/u/rocknroll)\ **Replies:** 2\ **Last updated:** [July 18, 2018, 2:16am UTC](https://discuss.elastic.co/t/filtered-and-unfiltered-outputs/140239 "2018-07-18T02:16:20Z")
I would like to be able to take a copy of each incoming Beats event and output directly to a file, then pass the events through a filter and output to Elasticsearch, i.e. Input-\>Output (file)-\>Filter-\>Output(Elasticsear… --- ## [S3 Input plugin - Settings to remember the last read line](https://discuss.elastic.co/t/s3-input-plugin-settings-to-remember-the-last-read-line/140204)
**Author:** [@Jay1](https://discuss.elastic.co/u/Jay1)\ **Replies:** 1\ **Last updated:** [July 18, 2018, 1:49am UTC](https://discuss.elastic.co/t/s3-input-plugin-settings-to-remember-the-last-read-line/140204 "2018-07-18T01:49:39Z")
I am using the s3 input plugin in logstash filter to read a streaming log file . This log file being an access log has a retention policy of 60 days on s3 . When i restart the logstash filter for different changes / ma… --- ## [HELP ! analyze unstructured data to be able to visualize them](https://discuss.elastic.co/t/help-analyze-unstructured-data-to-be-able-to-visualize-them/140373)
**Author:** [@Irving\_Hernandez](https://discuss.elastic.co/u/Irving_Hernandez)\ **Replies:** 1\ **Last updated:** [July 18, 2018, 12:54am UTC](https://discuss.elastic.co/t/help-analyze-unstructured-data-to-be-able-to-visualize-them/140373 "2018-07-18T00:54:55Z")
I want to analyze the file java.log are unstructured data which I would like to be able to obtain specific fields such as: error, java, idCatCatalog and ect. Intenete realizing a template loading it to Logstash but you d… --- ## [Logstash Java heap space](https://discuss.elastic.co/t/logstash-java-heap-space/140394)
**Author:** [@irom77](https://discuss.elastic.co/u/irom77)\ **Replies:** 1\ **Last updated:** [July 17, 2018, 11:53pm UTC](https://discuss.elastic.co/t/logstash-java-heap-space/140394 "2018-07-17T23:53:03Z")
My logstash 6.3 running in docker is failing, see below: logstash\_1 | java.lang.OutOfMemoryError: Java heap space logstash\_1 | Dumping heap to java\_pid1.hprof ... logstash\_1 | Unable to create java… --- ## [The given configuration is invalid. Reason: Expected one of #](https://discuss.elastic.co/t/the-given-configuration-is-invalid-reason-expected-one-of/140407)
**Author:** [@APJ](https://discuss.elastic.co/u/APJ)\ **Replies:** 2\ **Last updated:** [July 17, 2018, 10:51pm UTC](https://discuss.elastic.co/t/the-given-configuration-is-invalid-reason-expected-one-of/140407 "2018-07-17T22:51:02Z")
Hi all, I am following this link (Link) to display live weather data on Kibana. I used the logstash configuration explained in the pdf. Here is my logstash configuration input { exec { command =\> "curl --silent… --- ## [Decreased logstash/elasticsearch performance with load-balanced logstash nodes](https://discuss.elastic.co/t/decreased-logstash-elasticsearch-performance-with-load-balanced-logstash-nodes/138794)
**Author:** [@Kamen\_MacKay](https://discuss.elastic.co/u/Kamen_MacKay)\ **Replies:** 12\ **Last updated:** [July 17, 2018, 8:54pm UTC](https://discuss.elastic.co/t/decreased-logstash-elasticsearch-performance-with-load-balanced-logstash-nodes/138794 "2018-07-17T20:54:45Z")
I've been running an ELK stack on a single node for the past year to ingest our application server logs. The load is consistently quite high on this bare-metal box. I was recently given a smaller baremetal server to inst… --- ## [DIfferent Kinds of events from filebeat to Logstash, Assorting and Parsing](https://discuss.elastic.co/t/different-kinds-of-events-from-filebeat-to-logstash-assorting-and-parsing/140391)
**Author:** [@sdubbudu](https://discuss.elastic.co/u/sdubbudu)\ **Replies:** 2\ **Last updated:** [July 17, 2018, 8:26pm UTC](https://discuss.elastic.co/t/different-kinds-of-events-from-filebeat-to-logstash-assorting-and-parsing/140391 "2018-07-17T20:26:48Z")
Hello, I am trying to use filebeat to send two different types of events each from a different log to Logstash which does a different grok parse on each and send with a different id, but the same index to elasticsearc… --- ## [Large Logstash Instance](https://discuss.elastic.co/t/large-logstash-instance/140380)
**Author:** [@wrinehart](https://discuss.elastic.co/u/wrinehart)\ **Replies:** 0\ **Last updated:** [July 17, 2018, 7:16pm UTC](https://discuss.elastic.co/t/large-logstash-instance/140380 "2018-07-17T19:16:31Z")
Howdy, I inherited three fairly beefy servers for Logstash. They have 256 gigs of RAM and 56 CPUs. I know I can run a bunch of separate Logstash processes to fully utilize the servers (using Docker, etc.), but I was c… --- ## [Logstash Mutate Split doesn't work](https://discuss.elastic.co/t/logstash-mutate-split-doesnt-work/140227)
**Author:** [@deadPix3l](https://discuss.elastic.co/u/deadPix3l)\ **Replies:** 2\ **Last updated:** [July 17, 2018, 5:44pm UTC](https://discuss.elastic.co/t/logstash-mutate-split-doesnt-work/140227 "2018-07-17T17:44:22Z")
I'm stuck with 6.1.1 and cannot upgrade, sorry if this was known and fixed but ive been searching for hours for a workaround. using something similar to: filter { mutate { split =\> { "dottedField" =\> '.' } … --- ## [Logstash BigQuery Output into Partitioned Tables](https://discuss.elastic.co/t/logstash-bigquery-output-into-partitioned-tables/140353)
**Author:** [@elm](https://discuss.elastic.co/u/elm)\ **Replies:** 0\ **Last updated:** [July 17, 2018, 2:55pm UTC](https://discuss.elastic.co/t/logstash-bigquery-output-into-partitioned-tables/140353 "2018-07-17T14:55:39Z")
Hi, is there anyone who has been using https://www.elastic.co/guide/en/logstash/current/plugins-outputs-google\_bigquery.html with partitioned tables? The only way I see is to setup a table without a pattern / a static p… --- ## [How to move pipelines.yml to another location than conf folder?](https://discuss.elastic.co/t/how-to-move-pipelines-yml-to-another-location-than-conf-folder/140256)
**Author:** [@asp](https://discuss.elastic.co/u/asp)\ **Replies:** 2\ **Last updated:** [July 17, 2018, 2:02pm UTC](https://discuss.elastic.co/t/how-to-move-pipelines-yml-to-another-location-than-conf-folder/140256 "2018-07-17T14:02:07Z")
Hi, I am using multiple pipelines which are configured in conf/pipelines.yml. Is it possible to move that file to another location and read it from there? Thanks, Andreas --- ## [Concatenate using painless in Output](https://discuss.elastic.co/t/concatenate-using-painless-in-output/140317)
**Author:** [@saramali](https://discuss.elastic.co/u/saramali)\ **Replies:** 1\ **Last updated:** [July 17, 2018, 12:19pm UTC](https://discuss.elastic.co/t/concatenate-using-painless-in-output/140317 "2018-07-17T12:19:07Z")
I am trying to update the message body of an indexed log. To be specific I am trying to concatenate. I am getting no error on the logstash console yet the message body is not being updated. Here is the update part from … --- ## [Index Patterns: Please specify a default index pattern](https://discuss.elastic.co/t/index-patterns-please-specify-a-default-index-pattern/140310)
**Author:** [@Aziz\_Sahnoun](https://discuss.elastic.co/u/Aziz_Sahnoun)\ **Replies:** 7\ **Last updated:** [July 17, 2018, 11:38am UTC](https://discuss.elastic.co/t/index-patterns-please-specify-a-default-index-pattern/140310 "2018-07-17T11:38:49Z")
hello i'm using ELK on my virtual machine centos 7 using vSphere client i have some logs to visualise /var/log/secure i've connected my kibana via inginx and its working fine ! i've set this configuration for /etc/lo… --- ## [The difference between log4j and logback](https://discuss.elastic.co/t/the-difference-between-log4j-and-logback/140293)
**Author:** [@springCkWyy](https://discuss.elastic.co/u/springCkWyy)\ **Replies:** 0\ **Last updated:** [July 17, 2018, 9:29am UTC](https://discuss.elastic.co/t/the-difference-between-log4j-and-logback/140293 "2018-07-17T09:29:00Z")
Now, Logstash collect log by "log4j" and "logback"， But I need to compare them, and Do you have some materials for me? the different between them. --- ## [Using Redis Enterprise with Logstash](https://discuss.elastic.co/t/using-redis-enterprise-with-logstash/140284)
**Author:** [@intrepid1](https://discuss.elastic.co/u/intrepid1)\ **Replies:** 0\ **Last updated:** [July 17, 2018, 8:46am UTC](https://discuss.elastic.co/t/using-redis-enterprise-with-logstash/140284 "2018-07-17T08:46:09Z")
Hi there, Has anyone ever managed to make the Redis plugins work with Redis Enterprise? Kind regards, Matthew --- ## [Http Poller Looping](https://discuss.elastic.co/t/http-poller-looping/140273)
**Author:** [@HITI\_SINHA](https://discuss.elastic.co/u/HITI_SINHA)\ **Replies:** 0\ **Last updated:** [July 17, 2018, 7:56am UTC](https://discuss.elastic.co/t/http-poller-looping/140273 "2018-07-17T07:56:55Z")
Using an "Http Poller Plugin" in Logstash, how can we loop over it for multiple APIs. Basically, If I have to use http poller for say, 50 request objects hitting same API endpoint, what could be a smarter way than writin… --- ## [A short question of GROK to handle hex number](https://discuss.elastic.co/t/a-short-question-of-grok-to-handle-hex-number/139684)
**Author:** [@terrymu](https://discuss.elastic.co/u/terrymu)\ **Replies:** 7\ **Last updated:** [July 17, 2018, 7:35am UTC](https://discuss.elastic.co/t/a-short-question-of-grok-to-handle-hex-number/139684 "2018-07-17T07:35:11Z")
Hello every experts, I have met a short question. Here is a source input content, it is one function back trace of an issue, the content shows as below: 101c2f8f 1106f017 129570e2 1210da9a 1290fa12 10fb825d120e33ed 12… --- ## [Centos 7.5 with rsyslog 8.24 / logstash 6.3.0 - on logstash output always severity code 5 and facility code 1 - Other post found on but still not solved](https://discuss.elastic.co/t/centos-7-5-with-rsyslog-8-24-logstash-6-3-0-on-logstash-output-always-severity-code-5-and-facility-code-1-other-post-found-on-but-still-not-solved/137206)
**Author:** [@pge](https://discuss.elastic.co/u/pge)\ **Replies:** 1\ **Last updated:** [July 17, 2018, 7:29am UTC](https://discuss.elastic.co/t/centos-7-5-with-rsyslog-8-24-logstash-6-3-0-on-logstash-output-always-severity-code-5-and-facility-code-1-other-post-found-on-but-still-not-solved/137206 "2018-07-17T07:29:31Z")
Hello, I´m a ELK beginner and from one week I'm working on strange problem. I installed one ELK test infrastructure based on Centos 7 (7.5.1804) and we have to send data to logstash (logstash-6.3.0-1) usysing rsyslog (… --- ## [About Kafka output plugin](https://discuss.elastic.co/t/about-kafka-output-plugin/140260)
**Author:** [@seungtae](https://discuss.elastic.co/u/seungtae)\ **Replies:** 1\ **Last updated:** [July 17, 2018, 7:24am UTC](https://discuss.elastic.co/t/about-kafka-output-plugin/140260 "2018-07-17T07:24:52Z")
Hello every one. I'm using the Kafka output plugin to send data to Spark. My code is like this : output { stdout { codec =\> rubydebug } kafka { codec =\> rubydebug topic\_id =\> "toSpark" bootstrap\_servers =\>… --- ## [Issue using logstash-input-jms with IBM MQ 8](https://discuss.elastic.co/t/issue-using-logstash-input-jms-with-ibm-mq-8/140254)
**Author:** [@chris\_a.k](https://discuss.elastic.co/u/chris_a.k)\ **Replies:** 0\ **Last updated:** [July 17, 2018, 6:47am UTC](https://discuss.elastic.co/t/issue-using-logstash-input-jms-with-ibm-mq-8/140254 "2018-07-17T06:47:48Z")
Hello guys, I am currently using logstash-input-jms plugin with IBM MQ 8, the pipeline is starting successfully but not picking up the message from the queue logstash.version"=\>"6.3.1" logstash-input-jms (3.0.5) \[roo… --- ## [Drill down in kibana](https://discuss.elastic.co/t/drill-down-in-kibana/140030)
**Author:** [@Gokul\_15](https://discuss.elastic.co/u/Gokul_15)\ **Replies:** 2\ **Last updated:** [July 15, 2018, 7:56am UTC](https://discuss.elastic.co/t/drill-down-in-kibana/140030 "2018-07-15T07:56:43Z")
is drill down is possible in kibana??if so how?? drill down example:in pie chart if one field is selected it will extend the info in other charts ... is it all possible in kibana ??? --- ## [Many files in input path](https://discuss.elastic.co/t/many-files-in-input-path/138883)
**Author:** [@safhw91](https://discuss.elastic.co/u/safhw91)\ **Replies:** 3\ **Last updated:** [July 17, 2018, 5:35am UTC](https://discuss.elastic.co/t/many-files-in-input-path/138883 "2018-07-17T05:35:32Z")
input { file { path =\> \[ "${folder}\\log1.txt", "${folder}\\log2.txt" ... ... ... "${folder}\\log1000.txt" \] } } Is there any problem with this kind of configuration to have logstash put 100+ file locations as i… --- ## [Logstash constantly losing connection to Elastic](https://discuss.elastic.co/t/logstash-constantly-losing-connection-to-elastic/137737)
**Author:** [@safhw91](https://discuss.elastic.co/u/safhw91)\ **Replies:** 2\ **Last updated:** [July 17, 2018, 5:34am UTC](https://discuss.elastic.co/t/logstash-constantly-losing-connection-to-elastic/137737 "2018-07-17T05:34:33Z")
I've been doing parsing with Logstash on my new machine in the exact same way as I did in the past, but Logstash is constantly losing connection to Elastic this time ("Attempted to send bulk request but there are no livi… --- ## [Logstash config error 6.3.1](https://discuss.elastic.co/t/logstash-config-error-6-3-1/140009)
**Author:** [@logesh\_j](https://discuss.elastic.co/u/logesh_j)\ **Replies:** 6\ **Last updated:** [July 17, 2018, 12:13am UTC](https://discuss.elastic.co/t/logstash-config-error-6-3-1/140009 "2018-07-17T00:13:48Z")
Not able to ingest data using logstash. Error as : \[2018-07-13T16:01:15,722\]\[DEBUG\]\[logstash.config.source.local.configpathloader\] Skipping the following files while reading config since they don't match the specified … --- ## [Where is set @Timestamp](https://discuss.elastic.co/t/where-is-set-timestamp/140205)
**Author:** [@Sebastian\_Herrera](https://discuss.elastic.co/u/Sebastian_Herrera)\ **Replies:** 1\ **Last updated:** [July 16, 2018, 8:45pm UTC](https://discuss.elastic.co/t/where-is-set-timestamp/140205 "2018-07-16T20:45:06Z")
Hello, i am wondering to know where is the @timestamp value set. Because i have a lot of nodes sending logs to my logstash but not all the nodes have the same timezone, so i am wondering to know if i set the logstash nod… --- ## [Registering events in Drip via REST API with logstash](https://discuss.elastic.co/t/registering-events-in-drip-via-rest-api-with-logstash/140211)
**Author:** [@pjvv1](https://discuss.elastic.co/u/pjvv1)\ **Replies:** 0\ **Last updated:** [July 16, 2018, 7:55pm UTC](https://discuss.elastic.co/t/registering-events-in-drip-via-rest-api-with-logstash/140211 "2018-07-16T19:55:47Z")
Hi, I'd like to register some events of an client-server app in Drip (https://www.drip.com) by using Drip REST API but I've failed doing so. My app generates JSON log such as: { "logdate": "2018-07-16 10:57:49.2574", … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=719) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=721)