# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=807 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 808 --- ## [Logstash vs Filebeat Output Logs](https://discuss.elastic.co/t/logstash-vs-filebeat-output-logs/122904)
**Author:** [@rickyrome86](https://discuss.elastic.co/u/rickyrome86)\ **Replies:** 2\ **Last updated:** [March 7, 2018, 7:12pm UTC](https://discuss.elastic.co/t/logstash-vs-filebeat-output-logs/122904 "2018-03-07T19:12:33Z")
What is the best method to send logs to another environment? We are currently sending logs internally via log-stash to our elastic search, but would like to send the logs to another environment managed by a 3rd party. I … --- ## [If \_grokparsefailure break condition](https://discuss.elastic.co/t/if-grokparsefailure-break-condition/122918)
**Author:** [@pfseth](https://discuss.elastic.co/u/pfseth)\ **Replies:** 1\ **Last updated:** [March 7, 2018, 6:39pm UTC](https://discuss.elastic.co/t/if-grokparsefailure-break-condition/122918 "2018-03-07T18:39:49Z")
is there any way to break if conditions ? filter { grok { patterns\_dir =\> "/etc/logstash/file/patterns" match =\> { "message" =\> "%{COMMON\_HEADER}" } } if "\_grokparsefailure" in… --- ## [Duplicating the documents](https://discuss.elastic.co/t/duplicating-the-documents/122874)
**Author:** [@sivap](https://discuss.elastic.co/u/sivap)\ **Replies:** 1\ **Last updated:** [March 7, 2018, 4:14pm UTC](https://discuss.elastic.co/t/duplicating-the-documents/122874 "2018-03-07T16:14:03Z")
Hi Team, we are trying to index logs from a file to elasticsearch using fluentd when we use logstash\_format : true in our fluent config file we are getting duplicate documents in elasticsearch(viewing from kibana) if … --- ## [Ruby error when replacing @timestamp](https://discuss.elastic.co/t/ruby-error-when-replacing-timestamp/122917)
**Author:** [@Anabella\_Cristaldi](https://discuss.elastic.co/u/Anabella_Cristaldi)\ **Replies:** 1\ **Last updated:** [March 7, 2018, 4:11pm UTC](https://discuss.elastic.co/t/ruby-error-when-replacing-timestamp/122917 "2018-03-07T16:11:58Z")
Hi, I need to set the @timestamp field to date contained in another field of my log (instead of using the default indexation time) I tried to use the date filter pluging in combination with ruby plugin Using this cod… --- ## [Logstash grok to match the metricbeat type and add field](https://discuss.elastic.co/t/logstash-grok-to-match-the-metricbeat-type-and-add-field/122295)
**Author:** [@arun\_prasath1](https://discuss.elastic.co/u/arun_prasath1)\ **Replies:** 8\ **Last updated:** [March 7, 2018, 2:23pm UTC](https://discuss.elastic.co/t/logstash-grok-to-match-the-metricbeat-type-and-add-field/122295 "2018-03-07T14:23:55Z")
Hi Logstash Experts, I have metricbeat to logstash to Kibana pipeline. In the logstash, I have the below grok filter to parse java process. But it is not adding the field for the matched pattern. secondly, it is additio… --- ## [S3 input plugin not sending to Elastic Search](https://discuss.elastic.co/t/s3-input-plugin-not-sending-to-elastic-search/122520)
**Author:** [@Patrik\_Iselind](https://discuss.elastic.co/u/Patrik_Iselind)\ **Replies:** 2\ **Last updated:** [March 7, 2018, 1:52pm UTC](https://discuss.elastic.co/t/s3-input-plugin-not-sending-to-elastic-search/122520 "2018-03-07T13:52:25Z")
Hi, I have an issue where the S3 input plugin doesn't seem to propagate the processed data, at all. I can see that the files in my bucket are being processed: ... logstash\_1 | \[2018-03-05T13:09:36,102\]\[DEBUG\]\[log… --- ## [Slow down the intake from kafka to logstash](https://discuss.elastic.co/t/slow-down-the-intake-from-kafka-to-logstash/122863)
**Author:** [@hello4usharath](https://discuss.elastic.co/u/hello4usharath)\ **Replies:** 1\ **Last updated:** [March 7, 2018, 1:41pm UTC](https://discuss.elastic.co/t/slow-down-the-intake-from-kafka-to-logstash/122863 "2018-03-07T13:41:35Z")
Hi , Is there any way we limit the intake of message from kafka? Thanks Sharath --- ## [Logstash Date Filter Plugin: Field remains a string datatype](https://discuss.elastic.co/t/logstash-date-filter-plugin-field-remains-a-string-datatype/122533)
**Author:** [@Pimdebruijn](https://discuss.elastic.co/u/Pimdebruijn)\ **Replies:** 2\ **Last updated:** [March 7, 2018, 1:34pm UTC](https://discuss.elastic.co/t/logstash-date-filter-plugin-field-remains-a-string-datatype/122533 "2018-03-07T13:34:41Z")
Hi, I'm trying to import a CSV-file that contains a date field, into Elasticsearch. The date field contains dates that look like this: 5-3-2018 08:51:00 So the format is: dd-MM-YYYY HH:mm:ss (If I'm correct) I've i… --- ## [How to increase number of fields in index template?](https://discuss.elastic.co/t/how-to-increase-number-of-fields-in-index-template/122851)
**Author:** [@Vitaly\_il](https://discuss.elastic.co/u/Vitaly_il)\ **Replies:** 1\ **Last updated:** [March 7, 2018, 12:27pm UTC](https://discuss.elastic.co/t/how-to-increase-number-of-fields-in-index-template/122851 "2018-03-07T12:27:47Z")
I added total\_fields.limit into my default logstash index template: "settings" : { "number\_of\_shards" : 1, "number\_of\_replicas": 0, "index.mapping.total\_fields.limit" : "2000" } B… --- ## [Remove contents of field from another field](https://discuss.elastic.co/t/remove-contents-of-field-from-another-field/122849)
**Author:** [@xenoid](https://discuss.elastic.co/u/xenoid)\ **Replies:** 2\ **Last updated:** [March 7, 2018, 12:18pm UTC](https://discuss.elastic.co/t/remove-contents-of-field-from-another-field/122849 "2018-03-07T12:18:57Z")
Hello there, I need to remove the contents of some field from another field. This seems like an easy procedure, but sadly I couldn't get it to work. This is what I tried, but it didn't work: mutate { gsub =\> \["messag… --- ## [Remove array field if empty 6.1 version](https://discuss.elastic.co/t/remove-array-field-if-empty-6-1-version/122022)
**Author:** [@Eswar\_Kumar\_Musiboin](https://discuss.elastic.co/u/Eswar_Kumar_Musiboin)\ **Replies:** 3\ **Last updated:** [March 7, 2018, 12:05pm UTC](https://discuss.elastic.co/t/remove-array-field-if-empty-6-1-version/122022 "2018-03-07T12:05:49Z")
Please suggest how to remove array field if empty? I have tried below config : Logstash version is : 6.1.1 if \[tags\] == \[\] { mutate { remove\_field =\> \["tags"\] } } But getting logstash exception: \[2018-03-01T15:07… --- ## [Read Optional fields](https://discuss.elastic.co/t/read-optional-fields/122779)
**Author:** [@millerb1](https://discuss.elastic.co/u/millerb1)\ **Replies:** 1\ **Last updated:** [March 7, 2018, 10:46am UTC](https://discuss.elastic.co/t/read-optional-fields/122779 "2018-03-07T10:46:09Z")
We have installed File-Beats on multiple servers to harvest JBoss logs and save through the same Log-Stash pipeline into Elastic-Search. We have set the optional fields in File-Beats config and want to save the Channel a… --- ## [How to set charset in input, filter and output in logstash conf fille](https://discuss.elastic.co/t/how-to-set-charset-in-input-filter-and-output-in-logstash-conf-fille/122843)
**Author:** [@hzarrabi](https://discuss.elastic.co/u/hzarrabi)\ **Replies:** 0\ **Last updated:** [March 7, 2018, 8:19am UTC](https://discuss.elastic.co/t/how-to-set-charset-in-input-filter-and-output-in-logstash-conf-fille/122843 "2018-03-07T08:19:48Z")
Hi, I have in my logstash-plain.log this kind of log : \[2018-03-07T09:02:28,053\]\[WARN \]\[logstash.codecs.multiline\] Received an event that has a different character encoding than you configured. {:text=\>"\\\\u00002\\\\u0000… --- ## [Input-cloudwatch-logs skipping streams](https://discuss.elastic.co/t/input-cloudwatch-logs-skipping-streams/122708)
**Author:** [@amas](https://discuss.elastic.co/u/amas)\ **Replies:** 1\ **Last updated:** [March 7, 2018, 7:36am UTC](https://discuss.elastic.co/t/input-cloudwatch-logs-skipping-streams/122708 "2018-03-07T07:36:20Z")
Hi, I'm using Logstash with the input-cloudwatch-logs to fetch some historic log data (last 6 month) from Cloudwatch Logs. It seems like the plugin start fetching as it should, but skips some of the streams in the log-gr… --- ## [Multiple tables as input. Multiple pipelines?](https://discuss.elastic.co/t/multiple-tables-as-input-multiple-pipelines/122630)
**Author:** [@justin\_Kim](https://discuss.elastic.co/u/justin_Kim)\ **Replies:** 4\ **Last updated:** [March 7, 2018, 7:00am UTC](https://discuss.elastic.co/t/multiple-tables-as-input-multiple-pipelines/122630 "2018-03-07T07:00:26Z")
Hello, I'm a complete n00b at Logstash, so please excuse my ignorance. I have multple tables that I'd like to read using the JDBC input. They have different schemas, different purposes, etc. Consuming these data as an… --- ## [The final mapping would have more than 1 type](https://discuss.elastic.co/t/the-final-mapping-would-have-more-than-1-type/122565)
**Author:** [@jaas](https://discuss.elastic.co/u/jaas)\ **Replies:** 9\ **Last updated:** [March 7, 2018, 12:31am UTC](https://discuss.elastic.co/t/the-final-mapping-would-have-more-than-1-type/122565 "2018-03-07T00:31:18Z")
I write a template and want to load it manually, but failed. a message showed up : {"error": {"root\_cause":\[ {"type":"illegal\_argument\_exception","reason":"Rejecting mapping update to \[OC22HHTNQxaFWq2g\_XGKyA\] as the f… --- ## [Logstash Docker gives "Certificate verify failed" with Logstash Big Query Output](https://discuss.elastic.co/t/logstash-docker-gives-certificate-verify-failed-with-logstash-big-query-output/122778)
**Author:** [@iocgbht](https://discuss.elastic.co/u/iocgbht)\ **Replies:** 0\ **Last updated:** [March 6, 2018, 8:12pm UTC](https://discuss.elastic.co/t/logstash-docker-gives-certificate-verify-failed-with-logstash-big-query-output/122778 "2018-03-06T20:12:06Z")
Hi, I am trying to use Google Big Query Logstash output in a docker container. After starting the job I get the following errror: logstash\_1 | \[2018-03-06T19:57:58,505\]\[ERROR\]\[logstash.pipeline \] Error registering plug… --- ## [Grok TIMESTAMP\_ISO8601 doesnot capture TZ](https://discuss.elastic.co/t/grok-timestamp-iso8601-doesnot-capture-tz/122765)
**Author:** [@pfseth](https://discuss.elastic.co/u/pfseth)\ **Replies:** 1\ **Last updated:** [March 6, 2018, 9:06pm UTC](https://discuss.elastic.co/t/grok-timestamp-iso8601-doesnot-capture-tz/122765 "2018-03-06T21:06:51Z")
How can i parse TZ using grok 2018-03-05 21:29:31 -0500,188c3263-2519-4a8d-825c-4605c53b720e TIMESTAMP\_ISO8601 does not capture -0500 Grok Debugger ... { "TIMESTAMP\_ISO8601": \[ \[ "2018-03-05 21:29:31" … --- ## [Logstash 5.4 recreating the indices after deletion](https://discuss.elastic.co/t/logstash-5-4-recreating-the-indices-after-deletion/122583)
**Author:** [@dubul](https://discuss.elastic.co/u/dubul)\ **Replies:** 4\ **Last updated:** [March 6, 2018, 8:38pm UTC](https://discuss.elastic.co/t/logstash-5-4-recreating-the-indices-after-deletion/122583 "2018-03-06T20:38:55Z")
I have filebeats reading the log files on a remote server and shipping it to logstash on the same server. I have tried deleting the last 6 months old indices but I have noticed that indices are recreated on the elastic s… --- ## [XML to Elasticsearch via Logstash](https://discuss.elastic.co/t/xml-to-elasticsearch-via-logstash/120726)
**Author:** [@Errors](https://discuss.elastic.co/u/Errors)\ **Replies:** 46\ **Last updated:** [March 6, 2018, 7:40pm UTC](https://discuss.elastic.co/t/xml-to-elasticsearch-via-logstash/120726 "2018-03-06T19:40:52Z")
So I am trying to parse an XML file which is a few thousand lines long, and looks like this: \ \ **Author:** [@emwferm](https://discuss.elastic.co/u/emwferm)\ **Replies:** 1\ **Last updated:** [March 6, 2018, 6:39pm UTC](https://discuss.elastic.co/t/logstash-6-2-missing-library-netty-tcnative-linux-x86-64/120428 "2018-03-06T18:39:11Z") Hello: Trying out the new logstash 6.2 and we get the following error: \[2018-02-16T15:43:35,981\]\[DEBUG\]\[io.netty.util.internal.NativeLibraryLoader\] Unable to load the library 'netty\_tcnative\_linux\_x86\_64', trying other… --- ## [Logstash 6.2.2 JDBC extraction failing](https://discuss.elastic.co/t/logstash-6-2-2-jdbc-extraction-failing/122746)
**Author:** [@fabiofernandes](https://discuss.elastic.co/u/fabiofernandes)\ **Replies:** 0\ **Last updated:** [March 6, 2018, 4:17pm UTC](https://discuss.elastic.co/t/logstash-6-2-2-jdbc-extraction-failing/122746 "2018-03-06T16:17:00Z")
Hi, Our team was using Logstash 5.6.5 to extract a table from a MSSQL database. After updating to Logstash 6.2.2 the same config and SQL query stopped working. It gives the following error: \[WARN \] 2018-03-06 16:08:29.… --- ## [Documents missing after jdbc input to Elastic](https://discuss.elastic.co/t/documents-missing-after-jdbc-input-to-elastic/121166)
**Author:** [@johnjingle](https://discuss.elastic.co/u/johnjingle)\ **Replies:** 1\ **Last updated:** [March 6, 2018, 3:23pm UTC](https://discuss.elastic.co/t/documents-missing-after-jdbc-input-to-elastic/121166 "2018-03-06T15:23:29Z")
We have 50000000 records in SQL Server database and after using Logstash JDBC plugin to Elastic we are missing around 500 documents. Wondering what that might be. --- ## [Logstash websockets input generates \`force\_encoding\` exceptions](https://discuss.elastic.co/t/logstash-websockets-input-generates-force-encoding-exceptions/122723)
**Author:** [@Loek](https://discuss.elastic.co/u/Loek)\ **Replies:** 0\ **Last updated:** [March 6, 2018, 1:36pm UTC](https://discuss.elastic.co/t/logstash-websockets-input-generates-force-encoding-exceptions/122723 "2018-03-06T13:36:24Z")
Logstash: 6.2.2 Installed logstash-input-websocket via logstash-plugin Input: input { websocket { id =\> "meetup-rsvp-websocket" url =\> "http://stream.meetup.com/2/rsvps" } } Errors: \[2018-03-06T01:24:15,… --- ## [Parsing Alienvault xml logs](https://discuss.elastic.co/t/parsing-alienvault-xml-logs/122605)
**Author:** [@jgwentworth](https://discuss.elastic.co/u/jgwentworth)\ **Replies:** 2\ **Last updated:** [March 6, 2018, 12:43pm UTC](https://discuss.elastic.co/t/parsing-alienvault-xml-logs/122605 "2018-03-06T12:43:20Z")
I'm looking to parse out the specific fields indicated in each entry id tag. \\ \ \ **Author:** [@cawoodm](https://discuss.elastic.co/u/cawoodm)\ **Replies:** 1\ **Last updated:** [March 6, 2018, 12:33pm UTC](https://discuss.elastic.co/t/which-ports-for-logstash-and-filebeat/122715 "2018-03-06T12:33:07Z") I'm trying to understand which ports Logstash listens on when sending logs from FileBeat to Logstash. In the logstash.yml we see http.port: 9600-9700 descibed as "Bind address for the metrics REST endpoint". In the fil… --- ## [Cisco netflow and format of flow\_start\_msec](https://discuss.elastic.co/t/cisco-netflow-and-format-of-flow-start-msec/122720)
**Author:** [@adankert](https://discuss.elastic.co/u/adankert)\ **Replies:** 0\ **Last updated:** [March 6, 2018, 12:31pm UTC](https://discuss.elastic.co/t/cisco-netflow-and-format-of-flow-start-msec/122720 "2018-03-06T12:31:21Z")
When I receive netflow data from Cisco there seems to be an issue with the type of flow\_start\_msec. I assume this should be treated as "date". But I get error messages like this: "error"=\>{"type"=\>"mapper\_parsing\_excep… --- ## [Docker commands hanging with no response after logstash restart](https://discuss.elastic.co/t/docker-commands-hanging-with-no-response-after-logstash-restart/122717)
**Author:** [@Funnjy](https://discuss.elastic.co/u/Funnjy)\ **Replies:** 0\ **Last updated:** [March 6, 2018, 12:18pm UTC](https://discuss.elastic.co/t/docker-commands-hanging-with-no-response-after-logstash-restart/122717 "2018-03-06T12:18:33Z")
Hi, I'm running the elastic stack on docker and have encountered a problem for several times now. Occasionally, when I'm running "docker restart logstash" command, process hangs up and after that other docker commands ha… --- ## [Logstash's elasticsearch plugin scrolId error](https://discuss.elastic.co/t/logstashs-elasticsearch-plugin-scrolid-error/122711)
**Author:** [@Frederico\_Ferreira](https://discuss.elastic.co/u/Frederico_Ferreira)\ **Replies:** 0\ **Last updated:** [March 6, 2018, 12:00pm UTC](https://discuss.elastic.co/t/logstashs-elasticsearch-plugin-scrolid-error/122711 "2018-03-06T12:00:46Z")
Hi, i'm using the following configuration: input { elasticsearch { hosts =\> \["xxx.xxx.xxx.xxx"\] index =\> "index\_tmp" query =\> '{ "query": { "query\_string": { "query": "\*" } } }' size =\> 500 docinf… --- ## [Problem with Fake Log in Logshash](https://discuss.elastic.co/t/problem-with-fake-log-in-logshash/122685)
**Author:** [@just666](https://discuss.elastic.co/u/just666)\ **Replies:** 7\ **Last updated:** [March 6, 2018, 11:11am UTC](https://discuss.elastic.co/t/problem-with-fake-log-in-logshash/122685 "2018-03-06T11:11:15Z")
I install Elastic (no problem) I install Kibana (no problem) I run Logshtast with argument: ./logstash -f apache\_config.conf apache\_config.conf input { file { path =\> "/var/log/my\_apache\_logs/\*.log" } } filter { … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=806) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=808)