# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=235 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 236 --- ## [Logstash container stuck in a restart loop](https://discuss.elastic.co/t/logstash-container-stuck-in-a-restart-loop/271372)
**Author:** [@gorbroth](https://discuss.elastic.co/u/gorbroth)\ **Replies:** 0\ **Last updated:** [April 27, 2021, 12:40pm UTC](https://discuss.elastic.co/t/logstash-container-stuck-in-a-restart-loop/271372 "2021-04-27T12:40:47Z")
Hi, My new logstash docker container is stuck in a restart loop . Error in logs - Does it try to require a relative path? That's been removed in Ruby 1.9. uri:classloader:/META-INF/jruby.home/lib/ru… --- ## [Index data into Elasticsearch without setting trigger on database tables](https://discuss.elastic.co/t/index-data-into-elasticsearch-without-setting-trigger-on-database-tables/270821)
**Author:** [@e-fo](https://discuss.elastic.co/u/e-fo)\ **Replies:** 3\ **Last updated:** [April 27, 2021, 8:49am UTC](https://discuss.elastic.co/t/index-data-into-elasticsearch-without-setting-trigger-on-database-tables/270821 "2021-04-27T08:49:50Z")
Hi, Our company is a mobile game publisher, we need analyze our published games data from their database and we needs use elasticsearch and kibana for some analysis but the game developers who work with us only give us … --- ## [Transform usage - How to push event for all specific terms even if one of them is missing](https://discuss.elastic.co/t/transform-usage-how-to-push-event-for-all-specific-terms-even-if-one-of-them-is-missing/271326)
**Author:** [@vigneshr35](https://discuss.elastic.co/u/vigneshr35)\ **Replies:** 0\ **Last updated:** [April 27, 2021, 7:20am UTC](https://discuss.elastic.co/t/transform-usage-how-to-push-event-for-all-specific-terms-even-if-one-of-them-is-missing/271326 "2021-04-27T07:20:50Z")
Hi Team, I have a scenario. I have a transform created that groups all events of each day from the source index into 3 different status - Completed, Terminated and Retry. For example, if for 25th Apr 2021 we have Comple… --- ## [Logstash configuration best practices](https://discuss.elastic.co/t/logstash-configuration-best-practices/271051)
**Author:** [@Dzious](https://discuss.elastic.co/u/Dzious)\ **Replies:** 3\ **Last updated:** [April 27, 2021, 7:09am UTC](https://discuss.elastic.co/t/logstash-configuration-best-practices/271051 "2021-04-27T07:09:17Z")
Hi everyone, I would like to know a bit more about logstash best practices I need to setup a monitoring system for about 20 servers at the moment and more to come in the future. Onto these servers, i'll have multiple … --- ## [Grok: tag\_on\_failure is not working and has no effect](https://discuss.elastic.co/t/grok-tag-on-failure-is-not-working-and-has-no-effect/271320)
**Author:** [@meerlicht](https://discuss.elastic.co/u/meerlicht)\ **Replies:** 0\ **Last updated:** [April 27, 2021, 6:40am UTC](https://discuss.elastic.co/t/grok-tag-on-failure-is-not-working-and-has-no-effect/271320 "2021-04-27T06:40:34Z")
Hi there, I can't get my head around why this snippet successfully groks the message, but neither sets nor removes the tags. I've tried to set a custom tag and also removing all tags (as in the current snippet) grok { … --- ## [Using pre-defined grok patterns](https://discuss.elastic.co/t/using-pre-defined-grok-patterns/271216)
**Author:** [@ThreatInter](https://discuss.elastic.co/u/ThreatInter)\ **Replies:** 4\ **Last updated:** [April 27, 2021, 4:48am UTC](https://discuss.elastic.co/t/using-pre-defined-grok-patterns/271216 "2021-04-27T04:48:21Z")
Hi, community. I just discovered that logstash has pre-defined grok patterns, for firewalls, for example (here logstash-patterns-core/patterns at master · logstash-plugins/logstash-patterns-core · GitHub). But I can't u… --- ## [Workaround to Dynamic Index Names When Ussing ILM Policy](https://discuss.elastic.co/t/workaround-to-dynamic-index-names-when-ussing-ilm-policy/271311)
**Author:** [@scott\_stash](https://discuss.elastic.co/u/scott_stash)\ **Replies:** 1\ **Last updated:** [April 27, 2021, 4:40am UTC](https://discuss.elastic.co/t/workaround-to-dynamic-index-names-when-ussing-ilm-policy/271311 "2021-04-27T04:40:32Z")
I apologize if this has been asked a lot, but I did a lot of reading and I'm still not quite clear. For instance: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#\_writing\_to\_differen… --- ## [Methods to loadbalance on logstash from Input data stream to Filter section](https://discuss.elastic.co/t/methods-to-loadbalance-on-logstash-from-input-data-stream-to-filter-section/271292)
**Author:** [@Shreesh\_Narayanan](https://discuss.elastic.co/u/Shreesh_Narayanan)\ **Replies:** 2\ **Last updated:** [April 26, 2021, 9:09pm UTC](https://discuss.elastic.co/t/methods-to-loadbalance-on-logstash-from-input-data-stream-to-filter-section/271292 "2021-04-26T21:09:13Z")
I'm looking for methods to loadbalance on logstash or elasticsearch . The requirement is that i have decently large log files with thousands of lines, while logstash is perfectly processing the log file as expected, i wa… --- ## [Replace @timestamp with actual timestamp from log file with two different formats](https://discuss.elastic.co/t/replace-timestamp-with-actual-timestamp-from-log-file-with-two-different-formats/271264)
**Author:** [@kommineni24](https://discuss.elastic.co/u/kommineni24)\ **Replies:** 5\ **Last updated:** [April 26, 2021, 8:25pm UTC](https://discuss.elastic.co/t/replace-timestamp-with-actual-timestamp-from-log-file-with-two-different-formats/271264 "2021-04-26T20:25:38Z")
Hi, I am trying to index my mail-relay log files to Elastic search. All the log entries are being indexed into a field named message . @timestamp field shows the time the entry was indexed and not the timestamp from the… --- ## [Improving fingerprint filter performance](https://discuss.elastic.co/t/improving-fingerprint-filter-performance/270573)
**Author:** [@vasu01](https://discuss.elastic.co/u/vasu01)\ **Replies:** 4\ **Last updated:** [April 26, 2021, 5:12pm UTC](https://discuss.elastic.co/t/improving-fingerprint-filter-performance/270573 "2021-04-26T17:12:23Z")
We are using the logstash fingerprint filter to avoid duplicate data in elasticsearch. But the data ingestion is taking more time. For Example - A file having ~15000-20000 rows takes approx 2~3 hours to load. Only two f… --- ## [Hide elasticsearch superuser name and password in logstash conf](https://discuss.elastic.co/t/hide-elasticsearch-superuser-name-and-password-in-logstash-conf/271124)
**Author:** [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)\ **Replies:** 2\ **Last updated:** [April 26, 2021, 3:17pm UTC](https://discuss.elastic.co/t/hide-elasticsearch-superuser-name-and-password-in-logstash-conf/271124 "2021-04-26T15:17:11Z")
Hello all, The logstash (7.x) in the environment resides on the same server as filebeat. I am reluctant to mention superuser name and password info for elasticsearch in logstash.conf. Can I create a new user in Kibana … --- ## [Logstash healthcheck](https://discuss.elastic.co/t/logstash-healthcheck/271088)
**Author:** [@sulsulatoff](https://discuss.elastic.co/u/sulsulatoff)\ **Replies:** 2\ **Last updated:** [April 26, 2021, 1:43pm UTC](https://discuss.elastic.co/t/logstash-healthcheck/271088 "2021-04-26T13:43:42Z")
Our Logstash runs in a pod every 30 mins. Once in a while, it stops working and we need to restart the pod. Is there a way logstash can notify an external server that it's up and running (an http endpoint) --- ## [Csv with date column](https://discuss.elastic.co/t/csv-with-date-column/271255)
**Author:** [@Muhammad\_Faisal](https://discuss.elastic.co/u/Muhammad_Faisal)\ **Replies:** 0\ **Last updated:** [April 26, 2021, 12:37pm UTC](https://discuss.elastic.co/t/csv-with-date-column/271255 "2021-04-26T12:37:34Z")
hi all, i have below blocks of csv data....date comes in first line followed by csv data ...i need to bring date with csv data as a column to process it with timestamp. 20210409 0030 Counter node-3 … --- ## [Environmental variable in Logstash](https://discuss.elastic.co/t/environmental-variable-in-logstash/271063)
**Author:** [@stillfreem](https://discuss.elastic.co/u/stillfreem)\ **Replies:** 1\ **Last updated:** [April 26, 2021, 6:25am UTC](https://discuss.elastic.co/t/environmental-variable-in-logstash/271063 "2021-04-26T06:25:41Z")
Hi Folks, I use Logstash to send logs to a cloud SIEM. In particular I use an output plugin for the SIEM Vendor In order to authenticate towards the cloud Logstash presents id which is public and don't bother me and k… --- ## [Logstash is not starting in windows](https://discuss.elastic.co/t/logstash-is-not-starting-in-windows/270577)
**Author:** [@Rahul\_Ravichandran](https://discuss.elastic.co/u/Rahul_Ravichandran)\ **Replies:** 4\ **Last updated:** [April 26, 2021, 4:36am UTC](https://discuss.elastic.co/t/logstash-is-not-starting-in-windows/270577 "2021-04-26T04:36:00Z")
Hey Guys , I am using Logstash to send logs to kibana . I have configured logstash in windows instances with Two pipelines in my pipelines.yaml and this is the config that is present in pipelines.yml - pipeline.id… --- ## [Split of JSON array into multiple events in Kibana](https://discuss.elastic.co/t/split-of-json-array-into-multiple-events-in-kibana/271018)
**Author:** [@pedro\_gonzalez](https://discuss.elastic.co/u/pedro_gonzalez)\ **Replies:** 7\ **Last updated:** [April 25, 2021, 5:55pm UTC](https://discuss.elastic.co/t/split-of-json-array-into-multiple-events-in-kibana/271018 "2021-04-25T17:55:28Z")
Hi, I am trying to split into different events (logs) this log schema: "\_index": "cocacola", "\_type": "raw", "\_id": "CqwJ63MBEQS11DmXsDyRZl", "\_score": 1, "\_source": { … --- ## [Not sure if a field is nill, null or empty. Help please](https://discuss.elastic.co/t/not-sure-if-a-field-is-nill-null-or-empty-help-please/271123)
**Author:** [@stillfreem](https://discuss.elastic.co/u/stillfreem)\ **Replies:** 4\ **Last updated:** [April 24, 2021, 6:55pm UTC](https://discuss.elastic.co/t/not-sure-if-a-field-is-nill-null-or-empty-help-please/271123 "2021-04-24T18:55:29Z")
Hi All, It's about barracuda FW log with the following entry +02:00 Security XXX Block: type=FWD|proto=TCP|srcIF=eth0|srcIP=0.0.0.0|srcPort=58570|srcMAC=00:00:00:00:00:00|dstIP=0.0.0.0|dstPort=3002|dstService=exlm-agen… --- ## [Cluster Health Events FROM Elastic TO Arcsight](https://discuss.elastic.co/t/cluster-health-events-from-elastic-to-arcsight/271185)
**Author:** [@yaharin\_ben\_ayon](https://discuss.elastic.co/u/yaharin_ben_ayon)\ **Replies:** 0\ **Last updated:** [April 25, 2021, 11:30am UTC](https://discuss.elastic.co/t/cluster-health-events-from-elastic-to-arcsight/271185 "2021-04-25T11:30:41Z")
Hello, is there a way to import the cluster health logs or any logs FROM Elastic TO Arcsight with the Logstash Arcsight module or other way? Thank you! --- ## [Logstash cannot use docker metadata to build a dynamic index](https://discuss.elastic.co/t/logstash-cannot-use-docker-metadata-to-build-a-dynamic-index/271032)
**Author:** [@ktpktr0](https://discuss.elastic.co/u/ktpktr0)\ **Replies:** 3\ **Last updated:** [April 25, 2021, 6:23am UTC](https://discuss.elastic.co/t/logstash-cannot-use-docker-metadata-to-build-a-dynamic-index/271032 "2021-04-25T06:23:05Z")
I use filebeat to collect docker logs and add a label when the docker container starts. - type: docker enabled: true containers.ids: - '\*' tail\_files: true The following configuration is made in logstash … --- ## [How to filter when dynamic index has no value](https://discuss.elastic.co/t/how-to-filter-when-dynamic-index-has-no-value/271166)
**Author:** [@ktpktr0](https://discuss.elastic.co/u/ktpktr0)\ **Replies:** 3\ **Last updated:** [April 25, 2021, 6:19am UTC](https://discuss.elastic.co/t/how-to-filter-when-dynamic-index-has-no-value/271166 "2021-04-25T06:19:24Z")
I use the label in the docker metadata to build an index dynamically. When the label of some containers does not exist or the value is empty, I get two indexes at the same time. How can I solve this problem Logstash c… --- ## [Should A Full DLQ Queue Effect Logstash Performance Signifigantly?](https://discuss.elastic.co/t/should-a-full-dlq-queue-effect-logstash-performance-signifigantly/271036)
**Author:** [@scott\_stash](https://discuss.elastic.co/u/scott_stash)\ **Replies:** 7\ **Last updated:** [April 25, 2021, 3:49am UTC](https://discuss.elastic.co/t/should-a-full-dlq-queue-effect-logstash-performance-signifigantly/271036 "2021-04-25T03:49:08Z")
I'm sending logs from Filebeat -\> Logstash -\> ES with 7.10.2. I have enabled DLQ and set it to 50 MB's. Once something is in the DLQ I do not bother reprocessing it and I'm fine with the data loss. Once I start seeing… --- ## [Table column value update into existing index(pdf ingest) column](https://discuss.elastic.co/t/table-column-value-update-into-existing-index-pdf-ingest-column/271162)
**Author:** [@joseph-l.amalraj](https://discuss.elastic.co/u/joseph-l.amalraj)\ **Replies:** 0\ **Last updated:** [April 25, 2021, 1:40am UTC](https://discuss.elastic.co/t/table-column-value-update-into-existing-index-pdf-ingest-column/271162 "2021-04-25T01:40:23Z")
Hi, I am new to elasticsearch. I have set of pdf files, i did ingested through FSCrawler(index=pdfvalitate). so, i got details in ElasticSearch. Now, i have a Table in mySQL TMPLT\_META(tmpl\_id, eff\_frm\_date, doc\_number),… --- ## [(urgent) Converting a list of decimals to float in CSV](https://discuss.elastic.co/t/urgent-converting-a-list-of-decimals-to-float-in-csv/268772)
**Author:** [@shafeeka](https://discuss.elastic.co/u/shafeeka)\ **Replies:** 4\ **Last updated:** [April 24, 2021, 9:14am UTC](https://discuss.elastic.co/t/urgent-converting-a-list-of-decimals-to-float-in-csv/268772 "2021-04-24T09:14:11Z")
mutate { convert =\> { "Question\_Vector" =\> "float\_eu" } I am trying to push a CSV file to elasticsearch using logstash. One of the column contains a list of vectors and I am trying to conv… --- ## [Does logstash include some color stripper plug-ins?](https://discuss.elastic.co/t/does-logstash-include-some-color-stripper-plug-ins/270945)
**Author:** [@wajika](https://discuss.elastic.co/u/wajika)\ **Replies:** 4\ **Last updated:** [April 23, 2021, 2:20pm UTC](https://discuss.elastic.co/t/does-logstash-include-some-color-stripper-plug-ins/270945 "2021-04-23T14:20:40Z")
like Can clean the color character output by docker drive. --- ## [Multiple beats input ports allowed on Logstash configuration?](https://discuss.elastic.co/t/multiple-beats-input-ports-allowed-on-logstash-configuration/271095)
**Author:** [@omkarg81](https://discuss.elastic.co/u/omkarg81)\ **Replies:** 1\ **Last updated:** [April 23, 2021, 2:07pm UTC](https://discuss.elastic.co/t/multiple-beats-input-ports-allowed-on-logstash-configuration/271095 "2021-04-23T14:07:28Z")
I'm trying to achieve getting Logstash to listen to 2 different beats ports - one for filebeat and one for winlogbeat, by setting the conf file to be - input { beats { id =\> "winlogbeat\_plugin" port =\> 504… --- ## [Grok Pattern for date and time format "03-MAR-21 00:40:21"](https://discuss.elastic.co/t/grok-pattern-for-date-and-time-format-03-mar-21-0021/271014)
**Author:** [@Wilks](https://discuss.elastic.co/u/Wilks)\ **Replies:** 3\ **Last updated:** [April 23, 2021, 1:12am UTC](https://discuss.elastic.co/t/grok-pattern-for-date-and-time-format-03-mar-21-0021/271014 "2021-04-23T01:12:04Z")
This is likely a simple one but what's the grok pattern for this? I have gone through all the ones I could find but I cant get a match for the format 03-MAR-21 00:40:2 Currently I am using a string but need to change i… --- ## [Replace @timestamp with actual timestamp from log file](https://discuss.elastic.co/t/replace-timestamp-with-actual-timestamp-from-log-file/271022)
**Author:** [@kommineni24](https://discuss.elastic.co/u/kommineni24)\ **Replies:** 2\ **Last updated:** [April 22, 2021, 11:04pm UTC](https://discuss.elastic.co/t/replace-timestamp-with-actual-timestamp-from-log-file/271022 "2021-04-22T23:04:46Z")
Hi, I am trying to index my mail-relay log files to Elastic search. All the log entries are being indexed into a field named message . @timestamp field shows the time the entry was indexed and not the timestamp from the… --- ## [Logstash dissect and grok pattern matching issue](https://discuss.elastic.co/t/logstash-dissect-and-grok-pattern-matching-issue/270983)
**Author:** [@mohsin106](https://discuss.elastic.co/u/mohsin106)\ **Replies:** 6\ **Last updated:** [April 22, 2021, 8:38pm UTC](https://discuss.elastic.co/t/logstash-dissect-and-grok-pattern-matching-issue/270983 "2021-04-22T20:38:47Z")
Hi, I'm trying to use dissect and grok pattern matching together in order to get the desired field:value pair that I want. However, I don't understand why it's not working. I'm parsing the description field which can co… --- ## [Log Files are not getting generated in the defined path of logstash.yml](https://discuss.elastic.co/t/log-files-are-not-getting-generated-in-the-defined-path-of-logstash-yml/271003)
**Author:** [@Krishna\_Kumar](https://discuss.elastic.co/u/Krishna_Kumar)\ **Replies:** 0\ **Last updated:** [April 22, 2021, 6:32pm UTC](https://discuss.elastic.co/t/log-files-are-not-getting-generated-in-the-defined-path-of-logstash-yml/271003 "2021-04-22T18:32:09Z")
Hi All, I am a beginner in ELK. Could you please advise here. In one of the ec2 server we have installed Logstash in that we have mounted the EBS volume under /data path and we are providing this path under log.path in… --- ## [Log files are not getting generated in the defined path (logstash.yml)](https://discuss.elastic.co/t/log-files-are-not-getting-generated-in-the-defined-path-logstash-yml/271002)
**Author:** [@Clyton](https://discuss.elastic.co/u/Clyton)\ **Replies:** 0\ **Last updated:** [April 22, 2021, 6:28pm UTC](https://discuss.elastic.co/t/log-files-are-not-getting-generated-in-the-defined-path-logstash-yml/271002 "2021-04-22T18:28:00Z")
Hi All, In Debugging Settings of logstash.yml, I'm defining as below, log.level: info path.log: /var/log/logstash However, for some reason, the logs are getting generated in /usr/share/logstash/logs Besides, I hav… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=234) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=236)