# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=544 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 545 --- ## [How to delete index data based on date field range using logstash command line](https://discuss.elastic.co/t/how-to-delete-index-data-based-on-date-field-range-using-logstash-command-line/182011)
**Author:** [@Venkatesh\_Ravuri](https://discuss.elastic.co/u/Venkatesh_Ravuri)\ **Replies:** 0\ **Last updated:** [May 21, 2019, 1:28pm UTC](https://discuss.elastic.co/t/how-to-delete-index-data-based-on-date-field-range-using-logstash-command-line/182011 "2019-05-21T13:28:07Z")
I am trying to perform delta delete for old data based on date field range, I tried few options but not working. Conf file: input { s3 { bucket =\> "#####" region =\> "#####" prefix =\> "#####" } } filter { csv { … --- ## [Logstash - Multiple grok pattern not working together](https://discuss.elastic.co/t/logstash-multiple-grok-pattern-not-working-together/181825)
**Author:** [@achatterjee104](https://discuss.elastic.co/u/achatterjee104)\ **Replies:** 3\ **Last updated:** [May 21, 2019, 12:42pm UTC](https://discuss.elastic.co/t/logstash-multiple-grok-pattern-not-working-together/181825 "2019-05-21T12:42:17Z")
I am very new in using Logstash. I have two kinds of log, Pattern 1 : --2019-05-09 08:53:45.057 -INFO 11736 --- \[ntainer#1-0-C-1\] c.s.s.service.MessageLogServiceImpl : \[adc7fd862db5307a688817198046b284dbb12b9347bed… --- ## [Running logstash on all nodes in a production cluster](https://discuss.elastic.co/t/running-logstash-on-all-nodes-in-a-production-cluster/181924)
**Author:** [@Harsha606](https://discuss.elastic.co/u/Harsha606)\ **Replies:** 2\ **Last updated:** [May 21, 2019, 11:30am UTC](https://discuss.elastic.co/t/running-logstash-on-all-nodes-in-a-production-cluster/181924 "2019-05-21T11:30:07Z")
In my app server cluster i have around 60+ nodes, where logs of the applications are written under below directory structure on all the nodes /apps/project-name/services/environment/service-name/ and it contains below… --- ## [Attempted to resurrect connection to dead ES instance, but got an error. Manticore::ResolutionFailure](https://discuss.elastic.co/t/attempted-to-resurrect-connection-to-dead-es-instance-but-got-an-error-manticore-resolutionfailure/181981)
**Author:** [@Jeeth](https://discuss.elastic.co/u/Jeeth)\ **Replies:** 0\ **Last updated:** [May 21, 2019, 11:03am UTC](https://discuss.elastic.co/t/attempted-to-resurrect-connection-to-dead-es-instance-but-got-an-error-manticore-resolutionfailure/181981 "2019-05-21T11:03:16Z")
Hi Team, Using saas Elastic service, Have been pushing logs from logstash to Elastic service setup but since yesterday the data is not being uploaded. Im seeing the following error in logstash log files. Attempted to … --- ## [Pass Metadata Field via HTTP Plugin](https://discuss.elastic.co/t/pass-metadata-field-via-http-plugin/181957)
**Author:** [@salkonuhannovic](https://discuss.elastic.co/u/salkonuhannovic)\ **Replies:** 3\ **Last updated:** [May 21, 2019, 10:48am UTC](https://discuss.elastic.co/t/pass-metadata-field-via-http-plugin/181957 "2019-05-21T10:48:45Z")
Hi, I want to ingest data to logstash with following command: curl -XPUT 'mylogstashurl.com' -d 'hello' I get this data into logstash but unfortunjately the @metadata field is always empty. Is there any header i need… --- ## [Cloudwatch logs in Logstash](https://discuss.elastic.co/t/cloudwatch-logs-in-logstash/181975)
**Author:** [@aksadvance](https://discuss.elastic.co/u/aksadvance)\ **Replies:** 0\ **Last updated:** [May 21, 2019, 10:07am UTC](https://discuss.elastic.co/t/cloudwatch-logs-in-logstash/181975 "2019-05-21T10:07:06Z")
Hi, Need to integrate cloudwatch logs with Logstash. Tried cloudwatch input plugin for logstash but it seems mainly for metrics not for Cloudwatch logs. Please suggest how to feed cloudwatch logs into logstash. Do we… --- ## [Logstash fails to start](https://discuss.elastic.co/t/logstash-fails-to-start/181972)
**Author:** [@premkumar](https://discuss.elastic.co/u/premkumar)\ **Replies:** 0\ **Last updated:** [May 21, 2019, 10:05am UTC](https://discuss.elastic.co/t/logstash-fails-to-start/181972 "2019-05-21T10:05:34Z")
Hi Team, I just downloaded logstash (version 7.1.0) for linux and trying to start it. But it get failed with the following error. \[dev@devlinux bin\]$ ./logstash -f /home/dev/Desktop/LOGSTASH/logstash-simple.conf Sendi… --- ## [Help using logstash dissect filtering](https://discuss.elastic.co/t/help-using-logstash-dissect-filtering/181852)
**Author:** [@retroisbest](https://discuss.elastic.co/u/retroisbest)\ **Replies:** 2\ **Last updated:** [May 21, 2019, 7:27am UTC](https://discuss.elastic.co/t/help-using-logstash-dissect-filtering/181852 "2019-05-21T07:27:33Z")
Hi there, I'm trying to map my fields using dissect, my data has no common delimiter apart from spaces. My filter section in logstash config looks like this if \[type\] == "syslog" { dissect { mapping =\> { … --- ## [Geting work containce in msg](https://discuss.elastic.co/t/geting-work-containce-in-msg/181933)
**Author:** [@MamBarry](https://discuss.elastic.co/u/MamBarry)\ **Replies:** 0\ **Last updated:** [May 21, 2019, 7:24am UTC](https://discuss.elastic.co/t/geting-work-containce-in-msg/181933 "2019-05-21T07:24:47Z")
Hello, I try to get work containce in msg and then I like to affect this work. thanks. --- ## [Mongodb-input-logstash problem](https://discuss.elastic.co/t/mongodb-input-logstash-problem/181874)
**Author:** [@Aymen\_Ben\_Moussa](https://discuss.elastic.co/u/Aymen_Ben_Moussa)\ **Replies:** 0\ **Last updated:** [May 20, 2019, 4:41pm UTC](https://discuss.elastic.co/t/mongodb-input-logstash-problem/181874 "2019-05-20T16:41:44Z")
The problem with the plugin is that it does not see updated documents, for example i have a document that contains the field "status" , at this point i have my logstash pipeline already started and if the document statu… --- ## [No "New Line" in syslog - can logstash create new line based off of a match](https://discuss.elastic.co/t/no-new-line-in-syslog-can-logstash-create-new-line-based-off-of-a-match/181901)
**Author:** [@mdhw3uxhwh](https://discuss.elastic.co/u/mdhw3uxhwh)\ **Replies:** 2\ **Last updated:** [May 20, 2019, 9:26pm UTC](https://discuss.elastic.co/t/no-new-line-in-syslog-can-logstash-create-new-line-based-off-of-a-match/181901 "2019-05-20T21:26:18Z")
I have a crazy issue that really stems from Fortinet log forwarding jumbling syslog. It seems that the log forwarding is not generating new lines for each log entry. An example is: \<190\>date=2019-05-17,time=20:08:41,… --- ## [Can netflow be used to detect data loss in logstash pipelines](https://discuss.elastic.co/t/can-netflow-be-used-to-detect-data-loss-in-logstash-pipelines/181897)
**Author:** [@sushmamagesh](https://discuss.elastic.co/u/sushmamagesh)\ **Replies:** 0\ **Last updated:** [May 20, 2019, 8:26pm UTC](https://discuss.elastic.co/t/can-netflow-be-used-to-detect-data-loss-in-logstash-pipelines/181897 "2019-05-20T20:26:07Z")
Can netflow be used to detect data loss in logstash pipelines? Use case I am trying to solve: I see occasionally data loading stops due to network changes. Data loads from mysql to elastic via logstash pipelines. I am… --- ## [Custom S3 Output Endpoint](https://discuss.elastic.co/t/custom-s3-output-endpoint/181890)
**Author:** [@Colby\_Dozier](https://discuss.elastic.co/u/Colby_Dozier)\ **Replies:** 0\ **Last updated:** [May 20, 2019, 7:11pm UTC](https://discuss.elastic.co/t/custom-s3-output-endpoint/181890 "2019-05-20T19:11:18Z")
I'm running into a wall trying to figure out how to get rid of ".amazonaws.com" from the URL that is built from the Logstash S3 output plugin. It does not work in the environment I am in. I have tried many things such a… --- ## [Merge or combine two indexes to one based on common field (id)](https://discuss.elastic.co/t/merge-or-combine-two-indexes-to-one-based-on-common-field-id/178678)
**Author:** [@aldol](https://discuss.elastic.co/u/aldol)\ **Replies:** 6\ **Last updated:** [May 20, 2019, 7:08pm UTC](https://discuss.elastic.co/t/merge-or-combine-two-indexes-to-one-based-on-common-field-id/178678 "2019-05-20T19:08:36Z")
I have one index (index1) with columns "name" and "id" and the other index (index2) with columns "resume" and "id". "id" is the common field between two indexes. How can I merge or combine them together so I get one in… --- ## [IPTABLES](https://discuss.elastic.co/t/iptables/181879)
**Author:** [@TheSun](https://discuss.elastic.co/u/TheSun)\ **Replies:** 0\ **Last updated:** [May 20, 2019, 5:31pm UTC](https://discuss.elastic.co/t/iptables/181879 "2019-05-20T17:31:06Z")
\`indent preformatted text by 4 spaces\` --- ## [Remove chars from Date/String](https://discuss.elastic.co/t/remove-chars-from-date-string/181461)
**Author:** [@schneider](https://discuss.elastic.co/u/schneider)\ **Replies:** 3\ **Last updated:** [May 20, 2019, 5:27pm UTC](https://discuss.elastic.co/t/remove-chars-from-date-string/181461 "2019-05-20T17:27:23Z")
I want to remove "-03:00" from the input "2019 May 16 13:27:49 -03:00" The complet config is : input { udp { port =\> 5141 type =\> "syslog" tags =\> \["i… --- ## [Extract single bucket document using Logstash](https://discuss.elastic.co/t/extract-single-bucket-document-using-logstash/181860)
**Author:** [@ioannisnikolakis](https://discuss.elastic.co/u/ioannisnikolakis)\ **Replies:** 0\ **Last updated:** [May 20, 2019, 3:09pm UTC](https://discuss.elastic.co/t/extract-single-bucket-document-using-logstash/181860 "2019-05-20T15:09:37Z")
Hello Elasticsearch Community, I did a top hits aggregation, and I was wondering, can I somehow extract the document selected at each bucket (one document per bucket) using logstash ? I am asking prior to trying it beca… --- ## [How can I use Logstash pipeline in Elastic Cloud?](https://discuss.elastic.co/t/how-can-i-use-logstash-pipeline-in-elastic-cloud/181853)
**Author:** [@grad](https://discuss.elastic.co/u/grad)\ **Replies:** 1\ **Last updated:** [May 20, 2019, 2:50pm UTC](https://discuss.elastic.co/t/how-can-i-use-logstash-pipeline-in-elastic-cloud/181853 "2019-05-20T14:50:18Z")
Hi all... In Kibana settings there are 5 fine settings elasticsearch, kibana, logstash, beats and security. What is "Logstash Pipeline" configuration if Logstash is not present in the stack? Otherwise, how can I use t… --- ## [Date parsing problems](https://discuss.elastic.co/t/date-parsing-problems/181843)
**Author:** [@ta7643](https://discuss.elastic.co/u/ta7643)\ **Replies:** 4\ **Last updated:** [May 20, 2019, 2:18pm UTC](https://discuss.elastic.co/t/date-parsing-problems/181843 "2019-05-20T14:18:58Z")
Hi, I have a very strange behavior when parsing date. Does anybody understand this behavior? Here the config: input { stdin { codec =\> json} } filter { if \[last\_seen\]{ date { tag\_on\_failure =\> true match =\>… --- ## [What are logstash indicies starting with 00?](https://discuss.elastic.co/t/what-are-logstash-indicies-starting-with-00/181841)
**Author:** [@bauermalzwei](https://discuss.elastic.co/u/bauermalzwei)\ **Replies:** 2\ **Last updated:** [May 20, 2019, 2:35pm UTC](https://discuss.elastic.co/t/what-are-logstash-indicies-starting-with-00/181841 "2019-05-20T14:35:05Z")
Hi there, I've experimented with logstash last month and got it working fine. However, today when trying to set up curator I noticed that along with the regular indicies "logstash-2019.05.20" for some days logstash seem… --- ## [Kv split everything after ";"](https://discuss.elastic.co/t/kv-split-everything-after/180403)
**Author:** [@ghost3h](https://discuss.elastic.co/u/ghost3h)\ **Replies:** 9\ **Last updated:** [May 20, 2019, 1:53pm UTC](https://discuss.elastic.co/t/kv-split-everything-after/180403 "2019-05-20T13:53:57Z")
So I am trying to split the following data format examples: 'process\_count'=259;300;400; 'cpu'=3.8%;;; 'available'=27.29GiB;50;57; I essentially want my key pair to be name and first value, e.g "process\_count" =\> "… --- ## [Logstash not listening on 5044](https://discuss.elastic.co/t/logstash-not-listening-on-5044/181642)
**Author:** [@thadc](https://discuss.elastic.co/u/thadc)\ **Replies:** 3\ **Last updated:** [May 20, 2019, 1:17pm UTC](https://discuss.elastic.co/t/logstash-not-listening-on-5044/181642 "2019-05-20T13:17:35Z")
Hi, I have had filebeat/logstash/elasticsearch running successfully on (I'll call it) boxA for several weeks. Recently I had the need to set up on (I'll call it) boxB. I am using the same version of the elk components on… --- ## [Java Runtime Environment stopped when running logstash](https://discuss.elastic.co/t/java-runtime-environment-stopped-when-running-logstash/181814)
**Author:** [@Prashanth\_SP](https://discuss.elastic.co/u/Prashanth_SP)\ **Replies:** 0\ **Last updated:** [May 20, 2019, 12:42pm UTC](https://discuss.elastic.co/t/java-runtime-environment-stopped-when-running-logstash/181814 "2019-05-20T12:42:40Z")
Hi, We are running logstash with 4 RabbitMQ input plugins, 1 file plugin and 1 elastcisearch output plugin in a single pipeline. During the network issue, all the buffered data during that period pushed into rabbitmq qu… --- ## [LogStash Filter block, can we persist variable to next events to compare](https://discuss.elastic.co/t/logstash-filter-block-can-we-persist-variable-to-next-events-to-compare/181821)
**Author:** [@vijaykrishnach1](https://discuss.elastic.co/u/vijaykrishnach1)\ **Replies:** 1\ **Last updated:** [May 20, 2019, 1:16pm UTC](https://discuss.elastic.co/t/logstash-filter-block-can-we-persist-variable-to-next-events-to-compare/181821 "2019-05-20T13:16:56Z")
We have a requirement in LogStash filter logic to persist an extracted field to compare for upcoming events. Can you please suggest any options to cache / presisted queue / local variable concepts that we can use to achi… --- ## [Can I decode the values in Logstash](https://discuss.elastic.co/t/can-i-decode-the-values-in-logstash/181796)
**Author:** [@sjabiulla](https://discuss.elastic.co/u/sjabiulla)\ **Replies:** 2\ **Last updated:** [May 20, 2019, 11:34am UTC](https://discuss.elastic.co/t/can-i-decode-the-values-in-logstash/181796 "2019-05-20T11:34:14Z")
Hi, I have the key value pairs in the log message and I want to do data mapping before indexing the data to ElasticSearch. Sample Log Message 1: operation=1 timetaken=50 Sample Log Message 2: operation=2 timetaken=100 … --- ## [Logstash - Match metadata from filebeat](https://discuss.elastic.co/t/logstash-match-metadata-from-filebeat/181743)
**Author:** [@shortcommand](https://discuss.elastic.co/u/shortcommand)\ **Replies:** 1\ **Last updated:** [May 20, 2019, 10:57am UTC](https://discuss.elastic.co/t/logstash-match-metadata-from-filebeat/181743 "2019-05-20T10:57:38Z")
Hi, I've noticed filebeat sends some additional metadata for each generated event such as: host.hostname, log.file.path and host.os.family. Can I access them in logstash and create additional fields to my event, based… --- ## [Need to store the value in cache memory and compare while parse in logstash](https://discuss.elastic.co/t/need-to-store-the-value-in-cache-memory-and-compare-while-parse-in-logstash/181791)
**Author:** [@Ganesh2303](https://discuss.elastic.co/u/Ganesh2303)\ **Replies:** 0\ **Last updated:** [May 20, 2019, 10:29am UTC](https://discuss.elastic.co/t/need-to-store-the-value-in-cache-memory-and-compare-while-parse-in-logstash/181791 "2019-05-20T10:29:20Z")
HI Team, I have some tricky challenge, Anyone can help me on this concept, I have one restricted field and that will extract from my log. That keyword will not occur on every message. If my log contain restricted value… --- ## [Parse linux cron log](https://discuss.elastic.co/t/parse-linux-cron-log/181620)
**Author:** [@Semir](https://discuss.elastic.co/u/Semir)\ **Replies:** 2\ **Last updated:** [May 20, 2019, 10:23am UTC](https://discuss.elastic.co/t/parse-linux-cron-log/181620 "2019-05-20T10:23:34Z")
Hello, I'm a newbie in ELK. i'm trying to parse cron log. i'm using grokdebugger to test my grok pattern. my input is like this : May 12 06:01:01 bela-vtu182 run-parts(/etc/cron.hourly)\[28118\]: starting 0anacron May … --- ## [Logstash Grok pattern not working](https://discuss.elastic.co/t/logstash-grok-pattern-not-working/181317)
**Author:** [@shabhaskar](https://discuss.elastic.co/u/shabhaskar)\ **Replies:** 4\ **Last updated:** [May 20, 2019, 9:48am UTC](https://discuss.elastic.co/t/logstash-grok-pattern-not-working/181317 "2019-05-20T09:48:07Z")
hi There , I am completely new to ELK and currently learning to configure it for different logs, ELK version 7.0 while parsing the GClogs to logstash through fileBeat I see the pattrent match fails( please note same p… --- ## [Logstash : Multi Pipelines Performances](https://discuss.elastic.co/t/logstash-multi-pipelines-performances/181583)
**Author:** [@Oxyds](https://discuss.elastic.co/u/Oxyds)\ **Replies:** 4\ **Last updated:** [May 20, 2019, 7:49am UTC](https://discuss.elastic.co/t/logstash-multi-pipelines-performances/181583 "2019-05-20T07:49:22Z")
Hi, I am using Logstash to parse IIS logs and to ingest some metrics about the system of my servers. So I am using Filebeat to send the logs and Metricbeat to send the metrics. However,to allow the analysis of future new… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=543) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=545)