# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=273 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 274 --- ## [Only partial data getting deciphered using cipher filter](https://discuss.elastic.co/t/only-partial-data-getting-deciphered-using-cipher-filter/258129)
**Author:** [@Vishnu\_mk](https://discuss.elastic.co/u/Vishnu_mk)\ **Replies:** 0\ **Last updated:** [December 9, 2020, 1:38pm UTC](https://discuss.elastic.co/t/only-partial-data-getting-deciphered-using-cipher-filter/258129 "2020-12-09T13:38:06Z")
Hi Team, I want to decipher a cipher text using logstash. My client is using AES128 encryption. So in logstash I am using cipher filter to decipher the ciphertext. They have also provided me the key , but unfortunately… --- ## [Timestamp search from file](https://discuss.elastic.co/t/timestamp-search-from-file/257853)
**Author:** [@mihai.radulescu](https://discuss.elastic.co/u/mihai.radulescu)\ **Replies:** 4\ **Last updated:** [December 9, 2020, 9:42am UTC](https://discuss.elastic.co/t/timestamp-search-from-file/257853 "2020-12-09T09:42:41Z")
Hello, I have no experience with Elasticsearch, however I wish to setup an environment to be able to search through some large csv files. The import with logstash works, however I have two issues: timestamp scale is u… --- ## [Data is lost when outputting to file](https://discuss.elastic.co/t/data-is-lost-when-outputting-to-file/258089)
**Author:** [@lon](https://discuss.elastic.co/u/lon)\ **Replies:** 0\ **Last updated:** [December 9, 2020, 8:06am UTC](https://discuss.elastic.co/t/data-is-lost-when-outputting-to-file/258089 "2020-12-09T08:06:00Z")
\[Please excuse my poor English...\] If output the file to the same file with a line codec of the file output, the data will be lost. my logstash.conf input { beats { port =\> "5044" } } … --- ## [In my filebeat enable system module and application logs](https://discuss.elastic.co/t/in-my-filebeat-enable-system-module-and-application-logs/257985)
**Author:** [@prakash22](https://discuss.elastic.co/u/prakash22)\ **Replies:** 2\ **Last updated:** [December 9, 2020, 7:01am UTC](https://discuss.elastic.co/t/in-my-filebeat-enable-system-module-and-application-logs/257985 "2020-12-09T07:01:01Z")
I am sending these logs to logstash. But here both application, system logs are going to one index. But I want send to two indexes. This is logstash pipeline. input { beats { port =\> 5044 ssl =\> false } } output … --- ## [Logstash filter for firewall packet drops](https://discuss.elastic.co/t/logstash-filter-for-firewall-packet-drops/258042)
**Author:** [@hassan.rana](https://discuss.elastic.co/u/hassan.rana)\ **Replies:** 2\ **Last updated:** [December 8, 2020, 8:48pm UTC](https://discuss.elastic.co/t/logstash-filter-for-firewall-packet-drops/258042 "2020-12-08T20:48:03Z")
I am trying to parse logs from an aggregate syslog server to logstash. I have been successful in shipping the logs from the syslog server to logstash. I can see multiple fields from the log that are being setup like (hos… --- ## [Logstash monitoring with Metricbeat "logstash-xpack" module - data not collected and not sent to ES](https://discuss.elastic.co/t/logstash-monitoring-with-metricbeat-logstash-xpack-module-data-not-collected-and-not-sent-to-es/258006)
**Author:** [@Ilya\_Geller](https://discuss.elastic.co/u/Ilya_Geller)\ **Replies:** 7\ **Last updated:** [December 8, 2020, 6:18pm UTC](https://discuss.elastic.co/t/logstash-monitoring-with-metricbeat-logstash-xpack-module-data-not-collected-and-not-sent-to-es/258006 "2020-12-08T18:18:03Z")
Hi, I'm running version 7.7 of Logstash and Metricbeat, and I'm trying to setup stack monitoring using the metricbeat xpack modules. So far I've enabled the elasticsearch-xpack module and it works fine, ES and Kibana m… --- ## [Fortigate parsing logs](https://discuss.elastic.co/t/fortigate-parsing-logs/258028)
**Author:** [@sg\_tanoe](https://discuss.elastic.co/u/sg_tanoe)\ **Replies:** 0\ **Last updated:** [December 8, 2020, 4:42pm UTC](https://discuss.elastic.co/t/fortigate-parsing-logs/258028 "2020-12-08T16:42:28Z")
I would like to know how can I adapt log fields from a fortigate? In case, I want to get a result with a variety of field numbers over multiple lines. --- ## [Kafka monitoring using JMX input plugin](https://discuss.elastic.co/t/kafka-monitoring-using-jmx-input-plugin/257936)
**Author:** [@madhan0618](https://discuss.elastic.co/u/madhan0618)\ **Replies:** 0\ **Last updated:** [December 8, 2020, 5:33am UTC](https://discuss.elastic.co/t/kafka-monitoring-using-jmx-input-plugin/257936 "2020-12-08T05:33:05Z")
I read some tech notes on using jolokio jars to get logstash to consume JMX data from kafka and send it ES. Also came across the JMX input plugin. Installed and configured it and I am able to get the JMX metrics from ka… --- ## [Handling MQRFH2 header in logstash](https://discuss.elastic.co/t/handling-mqrfh2-header-in-logstash/258016)
**Author:** [@hassan2sn](https://discuss.elastic.co/u/hassan2sn)\ **Replies:** 0\ **Last updated:** [December 8, 2020, 3:10pm UTC](https://discuss.elastic.co/t/handling-mqrfh2-header-in-logstash/258016 "2020-12-08T15:10:10Z")
Hi Guys, Can you please suggest , how to handle incoming MQRFH2 header in logstash, this is going to the dead letter queue, whenever there is an MQRFH2 / usr folder header in request --- ## [LogStash - Upload using MySQL](https://discuss.elastic.co/t/logstash-upload-using-mysql/257604)
**Author:** [@nottyheadedboss](https://discuss.elastic.co/u/nottyheadedboss)\ **Replies:** 12\ **Last updated:** [December 8, 2020, 2:52pm UTC](https://discuss.elastic.co/t/logstash-upload-using-mysql/257604 "2020-12-08T14:52:52Z")
I am new to the world of ELK and trying to learn. But Logstash is proving to be a big hurdle. I tried loading data from a CSV file and that caused lot of issues and I have been unable to upload using CSV. So I tried a ne… --- ## [Communication between ruby and translate plugin in logstash filter](https://discuss.elastic.co/t/communication-between-ruby-and-translate-plugin-in-logstash-filter/257828)
**Author:** [@hzarrabi](https://discuss.elastic.co/u/hzarrabi)\ **Replies:** 4\ **Last updated:** [December 8, 2020, 1:01pm UTC](https://discuss.elastic.co/t/communication-between-ruby-and-translate-plugin-in-logstash-filter/257828 "2020-12-08T13:01:00Z")
Hi all, Do you have any idea or example how in ruby plugin (in logstash filter) invoke translate plugin and then collect and use the translated data again in ruby plugin? Thank. Best regards --- ## [Logstash filter error "mapper \[@version\] cannot be changed from type \[keyword\] to \[text\]"](https://discuss.elastic.co/t/logstash-filter-error-mapper-version-cannot-be-changed-from-type-keyword-to-text/257409)
**Author:** [@rrcoudian](https://discuss.elastic.co/u/rrcoudian)\ **Replies:** 1\ **Last updated:** [December 8, 2020, 12:05pm UTC](https://discuss.elastic.co/t/logstash-filter-error-mapper-version-cannot-be-changed-from-type-keyword-to-text/257409 "2020-12-08T12:05:48Z")
Greetings, and thank you in advance for your support! Could you please help us with the syntax of our ES 7.10 logstash.conf file to parse an application log file via filebeat that was tested on ES 6.6, but we are now us… --- ## [How often does logstash send data to google cloud storage?](https://discuss.elastic.co/t/how-often-does-logstash-send-data-to-google-cloud-storage/257973)
**Author:** [@iooi](https://discuss.elastic.co/u/iooi)\ **Replies:** 1\ **Last updated:** [December 8, 2020, 11:06am UTC](https://discuss.elastic.co/t/how-often-does-logstash-send-data-to-google-cloud-storage/257973 "2020-12-08T11:06:03Z")
Config pipeline.yml output { stdout { codec =\> rubydebug } google\_cloud\_storage { bucket =\> "my\_nginx" json\_key\_file =\> "/my/secret/path/credentials.json" temp\_directory… --- ## [How to change my data to kv in Logstash](https://discuss.elastic.co/t/how-to-change-my-data-to-kv-in-logstash/257449)
**Author:** [@Mick1](https://discuss.elastic.co/u/Mick1)\ **Replies:** 5\ **Last updated:** [December 8, 2020, 10:15am UTC](https://discuss.elastic.co/t/how-to-change-my-data-to-kv-in-logstash/257449 "2020-12-08T10:15:56Z")
The source data format cannot be changed First stroke： {value=12123, key=ABC2}, {value=2223, key=AB21C}, {value=1243, key=A4B1C}, {value=15223, key=AB35C}, {value=122123, key=A24BC}, {value=122213, key=AB33C}, {value=2… --- ## [Error while starting logstash](https://discuss.elastic.co/t/error-while-starting-logstash/257895)
**Author:** [@Lipi\_Lapsiwala](https://discuss.elastic.co/u/Lipi_Lapsiwala)\ **Replies:** 6\ **Last updated:** [December 8, 2020, 1:32am UTC](https://discuss.elastic.co/t/error-while-starting-logstash/257895 "2020-12-08T01:32:56Z")
I am trying to integrate Elastic stack (Logstash, ElasticSearch and Kibana) with IBM websphere liberty profile. I have successfully created and configured server.xml file in wlp and able to start and access the server. I… --- ## [How to parse an array of multi-line json objects as separate documents in ES?](https://discuss.elastic.co/t/how-to-parse-an-array-of-multi-line-json-objects-as-separate-documents-in-es/255230)
**Author:** [@reillye](https://discuss.elastic.co/u/reillye)\ **Replies:** 3\ **Last updated:** [December 7, 2020, 10:22pm UTC](https://discuss.elastic.co/t/how-to-parse-an-array-of-multi-line-json-objects-as-separate-documents-in-es/255230 "2020-12-07T22:22:41Z")
I have a use case where I am going to be receiving a new file every hour that is picked up by filebeat and then sent to logstash. The file will look like this: \[ { "Results": { "StartWeekSec": \[ 2000, 1980… --- ## [Multiple pipelines output to one cluster - only one of them picks up logs](https://discuss.elastic.co/t/multiple-pipelines-output-to-one-cluster-only-one-of-them-picks-up-logs/257877)
**Author:** [@Ilya\_Geller](https://discuss.elastic.co/u/Ilya_Geller)\ **Replies:** 4\ **Last updated:** [December 7, 2020, 8:38pm UTC](https://discuss.elastic.co/t/multiple-pipelines-output-to-one-cluster-only-one-of-them-picks-up-logs/257877 "2020-12-07T20:38:48Z")
Hi, I have a logstash node that has two pipelines that pick up different inputs and do separate things but both of them output to the same ES cluster into different indeces. Both pipelines have the following defined: … --- ## [Kafka Output fails to authenticate with kerberos](https://discuss.elastic.co/t/kafka-output-fails-to-authenticate-with-kerberos/257906)
**Author:** [@burkeg](https://discuss.elastic.co/u/burkeg)\ **Replies:** 0\ **Last updated:** [December 7, 2020, 7:57pm UTC](https://discuss.elastic.co/t/kafka-output-fails-to-authenticate-with-kerberos/257906 "2020-12-07T19:57:56Z")
I'm trying to write data to Kafka broker authenticated with Kerberos with GSSAPI and SASL. When I start the logstash agent it errors out with Receive Timeout from Kerberos but when i use kinit to get the ticket from kerb… --- ## [Modify All Logs Before Giving To Filter](https://discuss.elastic.co/t/modify-all-logs-before-giving-to-filter/257815)
**Author:** [@rknd](https://discuss.elastic.co/u/rknd)\ **Replies:** 1\ **Last updated:** [December 7, 2020, 7:50pm UTC](https://discuss.elastic.co/t/modify-all-logs-before-giving-to-filter/257815 "2020-12-07T19:50:48Z")
Hello everyone, Imagine that I have a codec plugin for input and my logs like this : 00:23 Lorem Ipsum 13:35 Lorem Ipsum 14:00 DATE-LINE 01/01/2020 15:43 Lorem Ipsum 20:20 Lorem Ipsum 00:15 Lorem Ipsum 13:10 DAT… --- ## ["destination.geo.country\_iso\_code" field with logstash geoip?](https://discuss.elastic.co/t/destination-geo-country-iso-code-field-with-logstash-geoip/257482)
**Author:** [@Camille](https://discuss.elastic.co/u/Camille)\ **Replies:** 4\ **Last updated:** [December 7, 2020, 3:59pm UTC](https://discuss.elastic.co/t/destination-geo-country-iso-code-field-with-logstash-geoip/257482 "2020-12-07T15:59:07Z")
Hi, is it possible to have to " destination.geo.country\_iso\_code" field with the geoip plugin in logstash ? I use it but similar output fields are "destination.geo.country\_code2" and "destination.geo.country\_code3" and… --- ## [Logstash stopped processing because of an error: (SystemExit) exit](https://discuss.elastic.co/t/logstash-stopped-processing-because-of-an-error-systemexit-exit/251719)
**Author:** [@nitin194](https://discuss.elastic.co/u/nitin194)\ **Replies:** 23\ **Last updated:** [December 7, 2020, 12:47pm UTC](https://discuss.elastic.co/t/logstash-stopped-processing-because-of-an-error-systemexit-exit/251719 "2020-12-07T12:47:55Z")
We are trying to index Nginx access and error log separately in Elasticsearch. for that we have created Filbeat and Logstash config as below. Below is our /etc/filebeat/filebeat.yml configuration filebeat.inputs: … --- ## [Ilm\_pattern not getting appended to rollover\_alias when index is created](https://discuss.elastic.co/t/ilm-pattern-not-getting-appended-to-rollover-alias-when-index-is-created/257835)
**Author:** [@Bhavrendra](https://discuss.elastic.co/u/Bhavrendra)\ **Replies:** 0\ **Last updated:** [December 7, 2020, 10:55am UTC](https://discuss.elastic.co/t/ilm-pattern-not-getting-appended-to-rollover-alias-when-index-is-created/257835 "2020-12-07T10:55:47Z")
I'm facing problem with ILM\_Pattern when i'm taking input to #Logstash from #Filebeat. The ilm\_pattern is not getting appended with ilm\_rollover\_alias in this case. Note: Same thing works fine if logstash is taking inp… --- ## [How to prevent Data Loss on Multipipeline Configuration](https://discuss.elastic.co/t/how-to-prevent-data-loss-on-multipipeline-configuration/257822)
**Author:** [@MARCO\_RAMBALDI](https://discuss.elastic.co/u/MARCO_RAMBALDI)\ **Replies:** 0\ **Last updated:** [December 7, 2020, 9:40am UTC](https://discuss.elastic.co/t/how-to-prevent-data-loss-on-multipipeline-configuration/257822 "2020-12-07T09:40:37Z")
Hi all, I have configured different pipelines (managed with a multipipeline configuration). Often, the one with the most data doesn't load all of them. For that I have added a specific DLQ . I also have enabled persist… --- ## [Different grok patterns for different log lines](https://discuss.elastic.co/t/different-grok-patterns-for-different-log-lines/257772)
**Author:** [@rojin](https://discuss.elastic.co/u/rojin)\ **Replies:** 1\ **Last updated:** [December 6, 2020, 3:50pm UTC](https://discuss.elastic.co/t/different-grok-patterns-for-different-log-lines/257772 "2020-12-06T15:50:03Z")
Hello. I have an application log file with 8 different patterns in it (8 different lines). Is it correct to use an array of match in grok to output all of them? if I use conditional expressions, only one pattern (for exa… --- ## [Custom field (filebeat) in condition in logstash filter](https://discuss.elastic.co/t/custom-field-filebeat-in-condition-in-logstash-filter/257761)
**Author:** [@arp220](https://discuss.elastic.co/u/arp220)\ **Replies:** 3\ **Last updated:** [December 6, 2020, 3:48pm UTC](https://discuss.elastic.co/t/custom-field-filebeat-in-condition-in-logstash-filter/257761 "2020-12-06T15:48:57Z")
I have this config in filebeat.yml: filebeat.inputs: - type: log enabled: true paths: - /var/log/nginx/access.log fields\_under\_root: true fields: service\_name: "nginx" ##### OutPut ####### output.logstas… --- ## [Multiline parsing error](https://discuss.elastic.co/t/multiline-parsing-error/257774)
**Author:** [@Muhammad\_Faisal](https://discuss.elastic.co/u/Muhammad_Faisal)\ **Replies:** 1\ **Last updated:** [December 6, 2020, 3:46pm UTC](https://discuss.elastic.co/t/multiline-parsing-error/257774 "2020-12-06T15:46:55Z")
hi all, i am using multiline to parse below log format...1 log entry is separated by either line which appears as "!-----New----------" or "------------!" ..... i am using below regex ,although its segregating log entri… --- ## [How to parse the json string](https://discuss.elastic.co/t/how-to-parse-the-json-string/257768)
**Author:** [@stwang](https://discuss.elastic.co/u/stwang)\ **Replies:** 1\ **Last updated:** [December 6, 2020, 11:44am UTC](https://discuss.elastic.co/t/how-to-parse-the-json-string/257768 "2020-12-06T11:44:27Z")
I have a log file, which the content like below {"log":"{\\"time\\":\\"2020-12-02 22:42:00,535\\", \\"a\_key\\":\\"a\_value\\"}\\n","stream":"stdout","time":"2020-12-02T22:42:05.58702969Z"} I want to parse the log field to json o… --- ## [Separating fields using grok with custom patterns](https://discuss.elastic.co/t/separating-fields-using-grok-with-custom-patterns/257765)
**Author:** [@rojin](https://discuss.elastic.co/u/rojin)\ **Replies:** 0\ **Last updated:** [December 6, 2020, 9:07am UTC](https://discuss.elastic.co/t/separating-fields-using-grok-with-custom-patterns/257765 "2020-12-06T09:07:56Z")
Hello! I have a problem while parsing my log file: I can not get the separated fields in stdout using ruby debug. The pattern is okay while using DevTools or the recommended debugger. also I have different log lines whic… --- ## [\[WARN \]\[logstash.outputs.elasticsearch\]\[main\] Marking url as dead](https://discuss.elastic.co/t/warn-logstash-outputs-elasticsearch-main-marking-url-as-dead/257752)
**Author:** [@Cristiane\_Marcarini](https://discuss.elastic.co/u/Cristiane_Marcarini)\ **Replies:** 2\ **Last updated:** [December 6, 2020, 10:01am UTC](https://discuss.elastic.co/t/warn-logstash-outputs-elasticsearch-main-marking-url-as-dead/257752 "2020-12-06T10:01:49Z")
What can I do in these situations? because the logs go up, but they don't appear in the kibana. \[2020-12-05T20:11:29,881\]\[WARN \]\[logstash.outputs.elasticsearch\]\[main\] Marking url as dead. Last error: \[LogStash::Outputs:… --- ## [Problem when adding fields to logstash](https://discuss.elastic.co/t/problem-when-adding-fields-to-logstash/257744)
**Author:** [@Gigazo1d](https://discuss.elastic.co/u/Gigazo1d)\ **Replies:** 5\ **Last updated:** [December 5, 2020, 7:57pm UTC](https://discuss.elastic.co/t/problem-when-adding-fields-to-logstash/257744 "2020-12-05T19:57:35Z")
Hi guys! There was a problem adding the add\_field via the mutate filter plugin. I do this: filter { if \[type\] == "syslog" { mutate { add\_field =\> \[ "received\_at", "%{@timestamp}" \] … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=272) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=274)