# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=234 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 235 --- ## [Do we have data loss when logstash is unable to send logs to haproxy as haproxy server is unavailable](https://discuss.elastic.co/t/do-we-have-data-loss-when-logstash-is-unable-to-send-logs-to-haproxy-as-haproxy-server-is-unavailable/269216)
**Author:** [@muralikrishna](https://discuss.elastic.co/u/muralikrishna)\ **Replies:** 6\ **Last updated:** [April 29, 2021, 4:03pm UTC](https://discuss.elastic.co/t/do-we-have-data-loss-when-logstash-is-unable-to-send-logs-to-haproxy-as-haproxy-server-is-unavailable/269216 "2021-04-29T16:03:09Z")
Hi All, I am looking for some information on data/log loss incase of my below situation. I have centralized management cluster to which my logstash will be connected with and i have logs coming from beat agents and sys… --- ## [Filebeat and multiline events](https://discuss.elastic.co/t/filebeat-and-multiline-events/271665)
**Author:** [@the3rdway](https://discuss.elastic.co/u/the3rdway)\ **Replies:** 0\ **Last updated:** [April 29, 2021, 2:42pm UTC](https://discuss.elastic.co/t/filebeat-and-multiline-events/271665 "2021-04-29T14:42:16Z")
I'm working with a virtual appliance that has Filebeat baked in. I have no control over the Filebeat configuration, which is forwarding multiple files of different formats. I can filter on source and grab the file that… --- ## [Unable to install specific logstash version in Ubuntu](https://discuss.elastic.co/t/unable-to-install-specific-logstash-version-in-ubuntu/271663)
**Author:** [@Souvik\_Das](https://discuss.elastic.co/u/Souvik_Das)\ **Replies:** 0\ **Last updated:** [April 29, 2021, 2:22pm UTC](https://discuss.elastic.co/t/unable-to-install-specific-logstash-version-in-ubuntu/271663 "2021-04-29T14:22:48Z")
I want to install logstash 7.11.1 in Ubuntu 18.04 but getting below error while trying to install: ubuntu@ip-xxx-xx-xx-xxx:~$ sudo apt-get install logstash=7.11.1 Reading package lists... Done Building dependency tree … --- ## [Configuration reloading can't be used with 'config.string' (-e)](https://discuss.elastic.co/t/configuration-reloading-cant-be-used-with-config-string-e/271576)
**Author:** [@PAVAN\_KUMAR\_REDDY\_GP](https://discuss.elastic.co/u/PAVAN_KUMAR_REDDY_GP)\ **Replies:** 1\ **Last updated:** [April 29, 2021, 1:10am UTC](https://discuss.elastic.co/t/configuration-reloading-cant-be-used-with-config-string-e/271576 "2021-04-29T01:10:00Z")
Hi Team, Getting an error " Configuration reloading can't be used with 'config.string' (-e)" while running logstash with below command. I tried testing the command in logstash version 7.6.2 and 7.9.2 . Can you please … --- ## [Extract multiple xml of multiline log](https://discuss.elastic.co/t/extract-multiple-xml-of-multiline-log/271560)
**Author:** [@E\_T\_N](https://discuss.elastic.co/u/E_T_N)\ **Replies:** 3\ **Last updated:** [April 29, 2021, 12:21am UTC](https://discuss.elastic.co/t/extract-multiple-xml-of-multiline-log/271560 "2021-04-29T00:21:36Z")
Hello. I have a log file with similar estructure: 05 Feb 2021 14:00:00,213 \[AAA-11\] INFO - Line A:\[0\] 05 Feb 2021 14:00:00,231 \[AAA-11\] INFO - Line2:\[0\] 05 Feb 2021 14:00:00,231 \[AAA-11\] INFO - Line3\[: 0\] 05 Feb 202… --- ## [Logstash IPV4 bind issue](https://discuss.elastic.co/t/logstash-ipv4-bind-issue/271303)
**Author:** [@edvrfn](https://discuss.elastic.co/u/edvrfn)\ **Replies:** 6\ **Last updated:** [April 28, 2021, 9:09pm UTC](https://discuss.elastic.co/t/logstash-ipv4-bind-issue/271303 "2021-04-28T21:09:52Z")
I have stopped/disabled firewalld and restarted logstash/Elasticsearch services but netstat only shows them listening only on IPv6. Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address … --- ## [Unable to Parse Meraki Flow Syslog Message with GROK and KV Filters in Logstash](https://discuss.elastic.co/t/unable-to-parse-meraki-flow-syslog-message-with-grok-and-kv-filters-in-logstash/271446)
**Author:** [@Khughes0516](https://discuss.elastic.co/u/Khughes0516)\ **Replies:** 4\ **Last updated:** [April 28, 2021, 9:03pm UTC](https://discuss.elastic.co/t/unable-to-parse-meraki-flow-syslog-message-with-grok-and-kv-filters-in-logstash/271446 "2021-04-28T21:03:53Z")
Hello, I am having a lot of trouble with trying to parse a Cisco Meraki flow syslog message in Logstash. A sample message looks like this: Apr 25 13:12:41 gateway 1619356361.230223408 ip\_flow\_end src=10.1.130.45 dst=3… --- ## [NESTED ATTRIBUTE RENAME NOT WORKING](https://discuss.elastic.co/t/nested-attribute-rename-not-working/271545)
**Author:** [@kamalgunda](https://discuss.elastic.co/u/kamalgunda)\ **Replies:** 3\ **Last updated:** [April 28, 2021, 8:30pm UTC](https://discuss.elastic.co/t/nested-attribute-rename-not-working/271545 "2021-04-28T20:30:27Z")
Hi All, I am not able to rename the nested attribute through logstash mutate/filter/ruby codes while loading the Elasticsearch index and input data is from database. Sample input data and logstash config file given bel… --- ## [Calculate the time difference between 2 xml log lines](https://discuss.elastic.co/t/calculate-the-time-difference-between-2-xml-log-lines/271458)
**Author:** [@roua.B](https://discuss.elastic.co/u/roua.B)\ **Replies:** 4\ **Last updated:** [April 28, 2021, 7:49pm UTC](https://discuss.elastic.co/t/calculate-the-time-difference-between-2-xml-log-lines/271458 "2021-04-28T19:49:02Z")
Hello everyone, I have an XML log file that looks like this: \ \ \ **Author:** [@Animesh\_Agarwal](https://discuss.elastic.co/u/Animesh_Agarwal)\ **Replies:** 2\ **Last updated:** [April 28, 2021, 6:03pm UTC](https://discuss.elastic.co/t/how-to-monitor-the-number-of-events-in-dead-letter-queue/271410 "2021-04-28T18:03:45Z") Hi Team, We have configured Dead Letter Queue(DLQ) in our Logstash layer. We want to be alerted when the number of events in DLQ increase beyond a certain limit. I tried going through the Logstash Monitoring docs, but… --- ## [\_dateparsefailure mapping date and time to @timestamp field](https://discuss.elastic.co/t/dateparsefailure-mapping-date-and-time-to-timestamp-field/271509)
**Author:** [@Wilks](https://discuss.elastic.co/u/Wilks)\ **Replies:** 2\ **Last updated:** [April 28, 2021, 4:17pm UTC](https://discuss.elastic.co/t/dateparsefailure-mapping-date-and-time-to-timestamp-field/271509 "2021-04-28T16:17:52Z")
I am getting a \_dateparsefailure when I try to map the date and time the log arrived to the @timestamp field. Below is my filter, if I comment out the attempt to map to @timestamp I dont get a \_dateparsefailure so it so… --- ## [Logstash stop to output when one kafka node is down](https://discuss.elastic.co/t/logstash-stop-to-output-when-one-kafka-node-is-down/271412)
**Author:** [@Travis](https://discuss.elastic.co/u/Travis)\ **Replies:** 11\ **Last updated:** [April 28, 2021, 4:08pm UTC](https://discuss.elastic.co/t/logstash-stop-to-output-when-one-kafka-node-is-down/271412 "2021-04-28T16:08:22Z")
Hello everybody, I have a 3 kafka nodes cluster Logstash has a kafka input and output to elasticsearch and syslog (I'm using logstash output isolator pattern). When one kafka node is down (kafka3 in this case), logstas… --- ## [Input on UDP converting to default UTF-8 but want Hex Strings](https://discuss.elastic.co/t/input-on-udp-converting-to-default-utf-8-but-want-hex-strings/271513)
**Author:** [@Yasho](https://discuss.elastic.co/u/Yasho)\ **Replies:** 0\ **Last updated:** [April 28, 2021, 12:52pm UTC](https://discuss.elastic.co/t/input-on-udp-converting-to-default-utf-8-but-want-hex-strings/271513 "2021-04-28T12:52:43Z")
Why is logstash converting input stream by default to UTF-8, ASCII, Unsigned Char, I want the data either converted full or do not touch it.. There is no codec to format the received data on UDP port. Example: This i… --- ## [Try to add a new field with timestamp in logstash](https://discuss.elastic.co/t/try-to-add-a-new-field-with-timestamp-in-logstash/271512)
**Author:** [@David\_Barat](https://discuss.elastic.co/u/David_Barat)\ **Replies:** 0\ **Last updated:** [April 28, 2021, 12:52pm UTC](https://discuss.elastic.co/t/try-to-add-a-new-field-with-timestamp-in-logstash/271512 "2021-04-28T12:52:39Z")
Hi everyone, I am pretty new in ELK world and I would like to parse csv file with logstash. I have tried to add a time field with timestamp (YYYY MM DD format), but I think there is an issue in the format I have made … --- ## [Editing date time filed in json filter output](https://discuss.elastic.co/t/editing-date-time-filed-in-json-filter-output/271136)
**Author:** [@sahere37](https://discuss.elastic.co/u/sahere37)\ **Replies:** 7\ **Last updated:** [April 28, 2021, 10:52am UTC](https://discuss.elastic.co/t/editing-date-time-filed-in-json-filter-output/271136 "2021-04-28T10:52:27Z")
Hi all I am using filebeat 7.8 which sends log to logstash 7.8. Each line of log is as following: {"req": { "service": ser1,"re\_date": "2020-09-10T09:14:38.479"}} and my logstash filter is as following: inpu… --- ## [Can we use aggregate filter to get the total no of times an IP appears in the logs?](https://discuss.elastic.co/t/can-we-use-aggregate-filter-to-get-the-total-no-of-times-an-ip-appears-in-the-logs/271486)
**Author:** [@Shreesh\_Narayanan](https://discuss.elastic.co/u/Shreesh_Narayanan)\ **Replies:** 0\ **Last updated:** [April 28, 2021, 9:02am UTC](https://discuss.elastic.co/t/can-we-use-aggregate-filter-to-get-the-total-no-of-times-an-ip-appears-in-the-logs/271486 "2021-04-28T09:02:37Z")
I have a requirement , where an IP or hostname appears frequently in logs , can we use aggregate filter to get this count of IP /Hostnames ? --- ## [I want to import aws cloudtrail eventTime through logstash](https://discuss.elastic.co/t/i-want-to-import-aws-cloudtrail-eventtime-through-logstash/271479)
**Author:** [@loanshark](https://discuss.elastic.co/u/loanshark)\ **Replies:** 0\ **Last updated:** [April 28, 2021, 7:47am UTC](https://discuss.elastic.co/t/i-want-to-import-aws-cloudtrail-eventtime-through-logstash/271479 "2021-04-28T07:47:57Z")
I want to import aws cloudtrail eventTime through logstash. Works well but fails to get eventTime. my logstash.conf input { s3 { bucket =\> "xxxxx" prefix =\> "xxxxx" sincedb\_path =\> "/etc/logstash/sincedb/… --- ## [How to remove special character on JSON file in logstash](https://discuss.elastic.co/t/how-to-remove-special-character-on-json-file-in-logstash/271462)
**Author:** [@mario\_kazela](https://discuss.elastic.co/u/mario_kazela)\ **Replies:** 1\ **Last updated:** [April 28, 2021, 5:06am UTC](https://discuss.elastic.co/t/how-to-remove-special-character-on-json-file-in-logstash/271462 "2021-04-28T05:06:16Z")
Hi, I'm new with ELK. Just want to ask, how to remove "\[" on the logstash? \[{"uuid": "8153990955613214174559836954673410970", "label": \["Phone", "Laptop", "Pc", "Computer", "Iphone", "Mobile Phone", "Electronics", "Cel… --- ## [Convert File, Group, Record and Unit Separator](https://discuss.elastic.co/t/convert-file-group-record-and-unit-separator/271452)
**Author:** [@bibhuWork](https://discuss.elastic.co/u/bibhuWork)\ **Replies:** 0\ **Last updated:** [April 28, 2021, 3:01am UTC](https://discuss.elastic.co/t/convert-file-group-record-and-unit-separator/271452 "2021-04-28T03:01:20Z")
Hi Friends I am trying to convert my message containing ^\\ , ^\] , ^^ and ^\_ into comma using mutate-\>gsub but its not working I am trying out \\u005E, \\u001E , \\u001F etc but no luck. Can you please help me out here. S… --- ## [What is Logstash regex engine?](https://discuss.elastic.co/t/what-is-logstash-regex-engine/271387)
**Author:** [@grumo35](https://discuss.elastic.co/u/grumo35)\ **Replies:** 2\ **Last updated:** [April 27, 2021, 3:10pm UTC](https://discuss.elastic.co/t/what-is-logstash-regex-engine/271387 "2021-04-27T15:10:08Z")
Hi, Just came across some regex in config files today and i cannot figure out which engine it is based on and documentation about the syntax i should use in my config. the thing i've stumbled across : if \[field\] !~ /… --- ## [How to change Array(list) to field with filter pipeline of Logstash?](https://discuss.elastic.co/t/how-to-change-array-list-to-field-with-filter-pipeline-of-logstash/271327)
**Author:** [@Farid\_N](https://discuss.elastic.co/u/Farid_N)\ **Replies:** 1\ **Last updated:** [April 27, 2021, 3:03pm UTC](https://discuss.elastic.co/t/how-to-change-array-list-to-field-with-filter-pipeline-of-logstash/271327 "2021-04-27T15:03:01Z")
Hi there I elaborate my issue with an instance: Imagine bellow log (The field "events" is an array , there are more fields in the log but I ignore them to write and just wrote down the field "events") "events": \[ … --- ## [Substitution text if value missing](https://discuss.elastic.co/t/substitution-text-if-value-missing/271301)
**Author:** [@lquin1978](https://discuss.elastic.co/u/lquin1978)\ **Replies:** 4\ **Last updated:** [April 27, 2021, 2:54pm UTC](https://discuss.elastic.co/t/substitution-text-if-value-missing/271301 "2021-04-27T14:54:17Z")
Our firewall is sending log messages to our ELK stack. Most of these messages contain a source and destination IP address.. but occasionally these values are not present. These messages are being forward via email and … --- ## [If field == IPv6 assign a different value](https://discuss.elastic.co/t/if-field-ipv6-assign-a-different-value/271397)
**Author:** [@stillfreem](https://discuss.elastic.co/u/stillfreem)\ **Replies:** 0\ **Last updated:** [April 27, 2021, 2:32pm UTC](https://discuss.elastic.co/t/if-field-ipv6-assign-a-different-value/271397 "2021-04-27T14:32:38Z")
I have the following configuration (partial output) # Check if SourceIP address is IPv4 or IPv6. In case SourceIP is IPv6 change its value #if \[SourceIP\] =~ ":" { mutate { rename =\> { "SourceIP" =\> "SourceIP… --- ## [Logstash is failing when running as a service due to logstash-core-plugin-api.gemspec error](https://discuss.elastic.co/t/logstash-is-failing-when-running-as-a-service-due-to-logstash-core-plugin-api-gemspec-error/271385)
**Author:** [@rohitarorait82](https://discuss.elastic.co/u/rohitarorait82)\ **Replies:** 1\ **Last updated:** [April 27, 2021, 2:26pm UTC](https://discuss.elastic.co/t/logstash-is-failing-when-running-as-a-service-due-to-logstash-core-plugin-api-gemspec-error/271385 "2021-04-27T14:26:23Z")
Hi Team, My logstash was working completely fine till yesterday and suddenly it stopped working, now when I am trying to restart it using systemctl start logstash command , I am getting below error. Check your env for… --- ## [Extract value from a multivalued field](https://discuss.elastic.co/t/extract-value-from-a-multivalued-field/271359)
**Author:** [@chriselk](https://discuss.elastic.co/u/chriselk)\ **Replies:** 3\ **Last updated:** [April 27, 2021, 2:16pm UTC](https://discuss.elastic.co/t/extract-value-from-a-multivalued-field/271359 "2021-04-27T14:16:22Z")
Hi, trying a simple stuff with logstash but could not get the expected result... I have a multivalued Field like: "PUB\_WEIGHT": \[ "3", "4", "6", "8" \] and in want to convert to Field\_1: 3 … --- ## [Date issue](https://discuss.elastic.co/t/date-issue/271322)
**Author:** [@sasvmware](https://discuss.elastic.co/u/sasvmware)\ **Replies:** 6\ **Last updated:** [April 27, 2021, 2:11pm UTC](https://discuss.elastic.co/t/date-issue/271322 "2021-04-27T14:11:46Z")
Log file date is on 26th but in elasticsearch date show as 25th why ? Logstash conf file. input { beats { port =\> 5044 } } filter { grok { match =\> { "message" =\> "%{TIMESTAMP\_ISO8601:timestamp}%{GREEDYDATA:lt\_… --- ## [How do I separate stdout and stderr logs of Logstash service?](https://discuss.elastic.co/t/how-do-i-separate-stdout-and-stderr-logs-of-logstash-service/271393)
**Author:** [@hvardhan](https://discuss.elastic.co/u/hvardhan)\ **Replies:** 0\ **Last updated:** [April 27, 2021, 2:03pm UTC](https://discuss.elastic.co/t/how-do-i-separate-stdout-and-stderr-logs-of-logstash-service/271393 "2021-04-27T14:03:53Z")
Logstash is being run using systemd. I have tried adding: StandardError=/ebs/logs/some\_error.log in logstash.service but it doesn't work. Also added this in pipelines.yml output { if \[level\] == "WARN" { … --- ## [From CloudTrail S3 SQS to Elastic via Logstash](https://discuss.elastic.co/t/from-cloudtrail-s3-sqs-to-elastic-via-logstash/271391)
**Author:** [@Dimitri\_Goldshtein](https://discuss.elastic.co/u/Dimitri_Goldshtein)\ **Replies:** 0\ **Last updated:** [April 27, 2021, 1:58pm UTC](https://discuss.elastic.co/t/from-cloudtrail-s3-sqs-to-elastic-via-logstash/271391 "2021-04-27T13:58:49Z")
Hi all ! I try get logs from my aws s3 via logstash ! I generate get logs from CloudTrail to s3 with format \*\*\*\*.json.gz in logstash i use this input file input { sqs { queue =\> "test\_log… --- ## [I can’t install logstash-plug](https://discuss.elastic.co/t/i-can-t-install-logstash-plug/271268)
**Author:** [@Dimitri\_Goldshtein](https://discuss.elastic.co/u/Dimitri_Goldshtein)\ **Replies:** 4\ **Last updated:** [April 27, 2021, 1:45pm UTC](https://discuss.elastic.co/t/i-can-t-install-logstash-plug/271268 "2021-04-27T13:45:13Z")
Hi all gurus ! I can’t install any plugin logstash ! For example this plugin : https://github.com/logstash-plugins/logstash-input-sqs If I run install to plug I get this error : usr/share/logstash/bin/logstash-plugi… --- ## [Parsing audit log SAP](https://discuss.elastic.co/t/parsing-audit-log-sap/271373)
**Author:** [@San9](https://discuss.elastic.co/u/San9)\ **Replies:** 0\ **Last updated:** [April 27, 2021, 12:41pm UTC](https://discuss.elastic.co/t/parsing-audit-log-sap/271373 "2021-04-27T12:41:20Z")
hello team, there was a need to analyze SAP audit logs. A filebeat was installed on the machines to transfer logs from to logstash. Format of audit logs - a file is created every day and data is written in one line (me… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=233) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=235)