# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=109 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 110 --- ## [Logstash got response code 401 contacting elasticsearch url](https://discuss.elastic.co/t/logstash-got-response-code-401-contacting-elasticsearch-url/317081)
**Author:** [@mammodde](https://discuss.elastic.co/u/mammodde)\ **Replies:** 6\ **Last updated:** [October 21, 2022, 8:06am UTC](https://discuss.elastic.co/t/logstash-got-response-code-401-contacting-elasticsearch-url/317081 "2022-10-21T08:06:38Z")
Hi everyone, i configured logstash pipeline and yml, from logs I can see that I receive something but when logstash tries to connect to elasticsearch it fails giving me \[2022-10-20T08:26:09,740\]\[WARN \]\[logstash.licensec… --- ## [Logstash\_s3\_output plugin failing with assumerole role\_arn](https://discuss.elastic.co/t/logstash-s3-output-plugin-failing-with-assumerole-role-arn/317167)
**Author:** [@pushpalatha](https://discuss.elastic.co/u/pushpalatha)\ **Replies:** 0\ **Last updated:** [October 21, 2022, 6:30am UTC](https://discuss.elastic.co/t/logstash-s3-output-plugin-failing-with-assumerole-role-arn/317167 "2022-10-21T06:30:41Z")
logstash pod is up and running with s3 plugin install with the image 6.8.23. But failing to upload logs to s3 bucket, due to below error \[ERROR\]\[logstash.outputs.s3 \] Uploading failed, retrying. {:exception=\>Aws::… --- ## [Strict execution ordering of Logstash output plugins](https://discuss.elastic.co/t/strict-execution-ordering-of-logstash-output-plugins/317048)
**Author:** [@gordonjlee](https://discuss.elastic.co/u/gordonjlee)\ **Replies:** 5\ **Last updated:** [October 20, 2022, 10:49pm UTC](https://discuss.elastic.co/t/strict-execution-ordering-of-logstash-output-plugins/317048 "2022-10-20T22:49:13Z")
I have 2 outputs from a Logstash instance, one output to Elasticsearch and one output to a Kafka topic that is consumed by a homegrown ms. In practice, I have seen instances where the ms consuming the Kafka topic receiv… --- ## [Logstash is disaster](https://discuss.elastic.co/t/logstash-is-disaster/317151)
**Author:** [@Nikolas1306](https://discuss.elastic.co/u/Nikolas1306)\ **Replies:** 1\ **Last updated:** [October 20, 2022, 8:49pm UTC](https://discuss.elastic.co/t/logstash-is-disaster/317151 "2022-10-20T20:49:33Z")
200% of cpu without pipelines and not produce log Who develops it is aware of its total unreliability? response: deny that Logstash has problems is not the solution in my opinion and better not to use it and try to … --- ## [How can I parse this date format into @timestamp?](https://discuss.elastic.co/t/how-can-i-parse-this-date-format-into-timestamp/316261)
**Author:** [@Roger\_Huang](https://discuss.elastic.co/u/Roger_Huang)\ **Replies:** 5\ **Last updated:** [October 20, 2022, 7:51pm UTC](https://discuss.elastic.co/t/how-can-i-parse-this-date-format-into-timestamp/316261 "2022-10-20T19:51:04Z")
Hi everyone, I am new to ELK and I cannot seems to get this right. Keep getting date parse error. I also wanted to put the \_time to @timestamp. For your kind advise please. 2022-06-30T13:14:40.558 Edgar.conf inpu… --- ## [Multiple Elastic Search Outputs for two different Azure event hubs as input in Logstash](https://discuss.elastic.co/t/multiple-elastic-search-outputs-for-two-different-azure-event-hubs-as-input-in-logstash/317129)
**Author:** [@Anudeep\_Konaboina](https://discuss.elastic.co/u/Anudeep_Konaboina)\ **Replies:** 3\ **Last updated:** [October 20, 2022, 6:09pm UTC](https://discuss.elastic.co/t/multiple-elastic-search-outputs-for-two-different-azure-event-hubs-as-input-in-logstash/317129 "2022-10-20T18:09:49Z")
I have to configure log stash to read from 2 different event hubs and write them to two different Elasticsearch index as shown below: input { azure\_event\_hubs { config\_mode =\> "advanced" threads =\> 8 d… --- ## [How to get date parsing target as a date nor string](https://discuss.elastic.co/t/how-to-get-date-parsing-target-as-a-date-nor-string/317097)
**Author:** [@sh.user](https://discuss.elastic.co/u/sh.user)\ **Replies:** 3\ **Last updated:** [October 20, 2022, 4:03pm UTC](https://discuss.elastic.co/t/how-to-get-date-parsing-target-as-a-date-nor-string/317097 "2022-10-20T16:03:36Z")
Hello, I need to parse Field1 as date . I use date filter but when i specify target , i get target as string not date ( by default the timestamp is the target but this is not my need). date { match =\> \["Field1", "dd/M… --- ## [Logstash sometimes ignoring datastream configuration in elasticsearch output](https://discuss.elastic.co/t/logstash-sometimes-ignoring-datastream-configuration-in-elasticsearch-output/317089)
**Author:** [@hti](https://discuss.elastic.co/u/hti)\ **Replies:** 1\ **Last updated:** [October 20, 2022, 10:21am UTC](https://discuss.elastic.co/t/logstash-sometimes-ignoring-datastream-configuration-in-elasticsearch-output/317089 "2022-10-20T10:21:06Z")
Hello, we are running logstash 8.4.0 with multiple pipelines outputting to elasticsearch. Most of the time this works fine, but sometimes logstash will ignore the datastream configuration on startup and tries to write … --- ## [Ca certification error](https://discuss.elastic.co/t/ca-certification-error/317092)
**Author:** [@Siva\_Priya](https://discuss.elastic.co/u/Siva_Priya)\ **Replies:** 0\ **Last updated:** [October 20, 2022, 9:46am UTC](https://discuss.elastic.co/t/ca-certification-error/317092 "2022-10-20T09:46:44Z")
I have installed Elasticsearch 7.14.1 .The Elasticsearch is working fine. Then i have deployed Logstash.The issue is logstash cannot communicate with Elasticsearch My elasticsearch.yaml file is apiVersion: elasticsearc… --- ## [Extract datatime in log and convert in datatime field?](https://discuss.elastic.co/t/extract-datatime-in-log-and-convert-in-datatime-field/317021)
**Author:** [@Nikolas1306](https://discuss.elastic.co/u/Nikolas1306)\ **Replies:** 2\ **Last updated:** [October 19, 2022, 11:06pm UTC](https://discuss.elastic.co/t/extract-datatime-in-log-and-convert-in-datatime-field/317021 "2022-10-19T23:06:32Z")
i ve this config in logstash filter { grok { match =\> \["message", "%{TIMESTAMP\_ISO8601:timestamp\_message}"\] } date { match =\> \["timestamp\_message","YYYY-MM-dd HH:mm:ss"\] target =\> "@timesta… --- ## [Logstash plugin use @logger in new class](https://discuss.elastic.co/t/logstash-plugin-use-logger-in-new-class/317034)
**Author:** [@tylersiemers](https://discuss.elastic.co/u/tylersiemers)\ **Replies:** 1\ **Last updated:** [October 19, 2022, 8:42pm UTC](https://discuss.elastic.co/t/logstash-plugin-use-logger-in-new-class/317034 "2022-10-19T20:42:34Z")
New to logstash plugins here. How do you pass the @logger reference into another class within a plugin? The example below but trying to call another class outside and logging cause a failure. class SecondClass @logger.… --- ## [Why default index created logstash-2022.01.01-000001](https://discuss.elastic.co/t/why-default-index-created-logstash-2022-01-01-000001/316632)
**Author:** [@connectgeeks](https://discuss.elastic.co/u/connectgeeks)\ **Replies:** 7\ **Last updated:** [October 19, 2022, 6:35pm UTC](https://discuss.elastic.co/t/why-default-index-created-logstash-2022-01-01-000001/316632 "2022-10-19T18:35:27Z")
I'm using filebeat for forwarding logs to logstash and creating multiple datastreams in elasticsearch. I noticed one index created with name "logstash-2022.01.01-000001" and storing some of data from different datasteams… --- ## [Update\_by\_query and](https://discuss.elastic.co/t/update-by-query-and/317020)
**Author:** [@reed](https://discuss.elastic.co/u/reed)\ **Replies:** 0\ **Last updated:** [October 19, 2022, 3:13pm UTC](https://discuss.elastic.co/t/update-by-query-and/317020 "2022-10-19T15:13:48Z")
Hi all, I have a problem with update\_by\_query in a pipeline but I noticed that the same query used into dev tools works fine: PIPELINE http { url =\> "http://crsinsightdev.icc.crifnet.com:9200/loyd-crs\_insight\_s1\_a… --- ## [Unable to connect to Elastic search with https from Logstash](https://discuss.elastic.co/t/unable-to-connect-to-elastic-search-with-https-from-logstash/316995)
**Author:** [@Pramod\_Kumar\_G](https://discuss.elastic.co/u/Pramod_Kumar_G)\ **Replies:** 1\ **Last updated:** [October 19, 2022, 2:40pm UTC](https://discuss.elastic.co/t/unable-to-connect-to-elastic-search-with-https-from-logstash/316995 "2022-10-19T14:40:38Z")
I am a new bie to ELK .I was able to start successfully elastic server and kibana.I was trying to start Logstash server but I am getting errors as below when run command logstash -f .\\config\\sample.conf my sample.conf … --- ## [Logstash problems with certs - docker-compose](https://discuss.elastic.co/t/logstash-problems-with-certs-docker-compose/316991)
**Author:** [@mammodde](https://discuss.elastic.co/u/mammodde)\ **Replies:** 1\ **Last updated:** [October 19, 2022, 12:28pm UTC](https://discuss.elastic.co/t/logstash-problems-with-certs-docker-compose/316991 "2022-10-19T12:28:37Z")
Hi everyone, I'm trying to configure logstash but apparently I am inserting the wrong certificate logstash.outputs.elasticsearch\]\[main\] Failed to perform request {:message=\>"Certificate for \ doesn't match any of th… --- ## [Why logstash does not create field on Kibana](https://discuss.elastic.co/t/why-logstash-does-not-create-field-on-kibana/316978)
**Author:** [@glaucoperucchi](https://discuss.elastic.co/u/glaucoperucchi)\ **Replies:** 0\ **Last updated:** [October 19, 2022, 9:36am UTC](https://discuss.elastic.co/t/why-logstash-does-not-create-field-on-kibana/316978 "2022-10-19T09:36:50Z")
Hello guys, I need a your help. I did a filter with grok, but when I go to kibana I did not see the field of the filter. The log: type=USER\_ACCT msg=audit(1666101680.397:1224): pid=665272 uid=0 auid=525000037 ses=27 … --- ## [WAF logs Configuration from s3 bucket to Elasticsearch Using logstash s3 Plugin](https://discuss.elastic.co/t/waf-logs-configuration-from-s3-bucket-to-elasticsearch-using-logstash-s3-plugin/315243)
**Author:** [@dharminfadia](https://discuss.elastic.co/u/dharminfadia)\ **Replies:** 2\ **Last updated:** [October 19, 2022, 9:32am UTC](https://discuss.elastic.co/t/waf-logs-configuration-from-s3-bucket-to-elasticsearch-using-logstash-s3-plugin/315243 "2022-10-19T09:32:46Z")
Hello Everyone I am trying to Configuring WAF 2.0 logs from s3 Bucket to elasticsearch but I am facing some erroorm please Helpme for the same. My Current Logstash configuration input { s3 { access\_key\_id =\> "myid" … --- ## [Logstash to convert scientific notation to float](https://discuss.elastic.co/t/logstash-to-convert-scientific-notation-to-float/316231)
**Author:** [@the\_elkguy](https://discuss.elastic.co/u/the_elkguy)\ **Replies:** 7\ **Last updated:** [October 19, 2022, 9:25am UTC](https://discuss.elastic.co/t/logstash-to-convert-scientific-notation-to-float/316231 "2022-10-19T09:25:54Z")
Hello, I'm using logstash ruby filter to convert scientific notation from string to float which seem to be not working. I have used GROK to extract the scientific notation (1.989e-04) from the message and now converting… --- ## [How to rename a field with regex](https://discuss.elastic.co/t/how-to-rename-a-field-with-regex/316972)
**Author:** [@Saeedeh\_Moghimi](https://discuss.elastic.co/u/Saeedeh_Moghimi)\ **Replies:** 1\ **Last updated:** [October 19, 2022, 9:11am UTC](https://discuss.elastic.co/t/how-to-rename-a-field-with-regex/316972 "2022-10-19T09:11:05Z")
I have a log like this: {"A.amazon.aws.requestedParameters":{"testlabel1":"testvalue1"}} {"B.amazon.aws.requestedParameters":{"testlabel1":"testvalue1"}} ... and I want to change the field name to something like this: A… --- ## [Remove field with pattern using mutate](https://discuss.elastic.co/t/remove-field-with-pattern-using-mutate/316966)
**Author:** [@yuswanul](https://discuss.elastic.co/u/yuswanul)\ **Replies:** 1\ **Last updated:** [October 19, 2022, 9:08am UTC](https://discuss.elastic.co/t/remove-field-with-pattern-using-mutate/316966 "2022-10-19T09:08:00Z")
Hi there, i want to remove some field based on pattern. for example, i have some field like: iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.2 iso.org.dod.internet.mgmt.mib-2.interfaces.ifEntry.3 and i want to del… --- ## [Duplicated data in index](https://discuss.elastic.co/t/duplicated-data-in-index/316863)
**Author:** [@INS](https://discuss.elastic.co/u/INS)\ **Replies:** 2\ **Last updated:** [October 19, 2022, 8:26am UTC](https://discuss.elastic.co/t/duplicated-data-in-index/316863 "2022-10-19T08:26:43Z")
Hi I have a problem that sometime during uploading to elastic data with eventType number has a difrent date then it has in the file, This is suspicious that metadata has been changed somewhere, I guess that logstash are… --- ## [CSV file input renew interval](https://discuss.elastic.co/t/csv-file-input-renew-interval/316952)
**Author:** [@Julian\_Tang](https://discuss.elastic.co/u/Julian_Tang)\ **Replies:** 0\ **Last updated:** [October 19, 2022, 6:40am UTC](https://discuss.elastic.co/t/csv-file-input-renew-interval/316952 "2022-10-19T06:40:22Z")
Hi I have a question ,about a CSV file , I adds additional row , 1.what time well renew this data ? 2.if I need to change interval , how to do? 3.or another better choice to update this? input { file { path =\> … --- ## [Using if else in logstash filter](https://discuss.elastic.co/t/using-if-else-in-logstash-filter/316853)
**Author:** [@yuswanul](https://discuss.elastic.co/u/yuswanul)\ **Replies:** 8\ **Last updated:** [October 19, 2022, 5:13am UTC](https://discuss.elastic.co/t/using-if-else-in-logstash-filter/316853 "2022-10-19T05:13:50Z")
Hi everyone, i'd like to ask, is it possible to use OR operator in if else statement in logstash filter? so, i want to delete the event that has value "VoIP-Null0", "Null0", and "Loopback0" in data.ifDescr field. i … --- ## [Import CSV date format time difference](https://discuss.elastic.co/t/import-csv-date-format-time-difference/316907)
**Author:** [@Julian\_Tang](https://discuss.elastic.co/u/Julian_Tang)\ **Replies:** 6\ **Last updated:** [October 18, 2022, 9:50pm UTC](https://discuss.elastic.co/t/import-csv-date-format-time-difference/316907 "2022-10-18T21:50:00Z")
Hi I try to import a CSV file in logstash. but I have problem, the csv file first is date and time, I use separator to "timestamp", but it + 8 hours (my time zone is +8) so I how to set time don't +8 ? input { file… --- ## [Does logstash support zip format logs from s3 bucket](https://discuss.elastic.co/t/does-logstash-support-zip-format-logs-from-s3-bucket/316877)
**Author:** [@shrikantgulia](https://discuss.elastic.co/u/shrikantgulia)\ **Replies:** 7\ **Last updated:** [October 18, 2022, 6:20pm UTC](https://discuss.elastic.co/t/does-logstash-support-zip-format-logs-from-s3-bucket/316877 "2022-10-18T18:20:38Z")
Hi Team, Just want to know does logstash support the integration of logs from s3 bucket which are in .zip format? Regards --- ## [Ingest already created doc in pubsub to elk using logstash](https://discuss.elastic.co/t/ingest-already-created-doc-in-pubsub-to-elk-using-logstash/316924)
**Author:** [@Dennis\_John](https://discuss.elastic.co/u/Dennis_John)\ **Replies:** 0\ **Last updated:** [October 18, 2022, 5:00pm UTC](https://discuss.elastic.co/t/ingest-already-created-doc-in-pubsub-to-elk-using-logstash/316924 "2022-10-18T17:00:09Z")
Hi, I am using microservices and part of the job by microservices is to create a doc with index and doc id and other fields that will be pushed to pubsub. Currently, we are ingesting to elk by pulling messages from GCP… --- ## [Logstash hardware requirements](https://discuss.elastic.co/t/logstash-hardware-requirements/316851)
**Author:** [@perezdev](https://discuss.elastic.co/u/perezdev)\ **Replies:** 1\ **Last updated:** [October 18, 2022, 1:55pm UTC](https://discuss.elastic.co/t/logstash-hardware-requirements/316851 "2022-10-18T13:55:32Z")
Hello, I'm going to install logstash in a Linux VM. I'm trying to find out the best hardware requirements for this VM so that logstash could handle the amount of data expected as smooth as possible. The amount of data … --- ## [How to specify an UNC-Path (network share) for logstash file input plugin](https://discuss.elastic.co/t/how-to-specify-an-unc-path-network-share-for-logstash-file-input-plugin/316900)
**Author:** [@simon137](https://discuss.elastic.co/u/simon137)\ **Replies:** 0\ **Last updated:** [October 18, 2022, 1:29pm UTC](https://discuss.elastic.co/t/how-to-specify-an-unc-path-network-share-for-logstash-file-input-plugin/316900 "2022-10-18T13:29:16Z")
Hello, I tried specifying a UNC-Path for the "path"-Setting of the logstash file input plugin and unfortunately it doesn't work. Is this a known issue or does someone here know how to get this to work on a windows masc… --- ## [Inode change without reason](https://discuss.elastic.co/t/inode-change-without-reason/316898)
**Author:** [@Victor\_Caetano](https://discuss.elastic.co/u/Victor_Caetano)\ **Replies:** 0\ **Last updated:** [October 18, 2022, 1:23pm UTC](https://discuss.elastic.co/t/inode-change-without-reason/316898 "2022-10-18T13:23:37Z")
I Have an ELK stack stablished using the following workflow A DB2 server generating logs in .txt format, a FTP for ingest those logs into a S3 bucket in my AWS environment Than a s3fs-fuse mount system, using this S3 b… --- ## [Writing events to different indexes through a variable](https://discuss.elastic.co/t/writing-events-to-different-indexes-through-a-variable/316814)
**Author:** [@Aleksei](https://discuss.elastic.co/u/Aleksei)\ **Replies:** 7\ **Last updated:** [October 18, 2022, 10:19am UTC](https://discuss.elastic.co/t/writing-events-to-different-indexes-through-a-variable/316814 "2022-10-18T10:19:22Z")
Hi everyone I did not find a suitable topic in the community, I decided to create a topic. Has anyone tried this configuration ? output { \*\* \*\* elasticsearch { \*\* \*\* index =\> "%{\[some\_field\]\[sub\_field\]}-… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=108) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=110)