# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=163 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 164 --- ## [DataBase Connection](https://discuss.elastic.co/t/database-connection/296167)
**Author:** [@Harsh1999](https://discuss.elastic.co/u/Harsh1999)\ **Replies:** 1\ **Last updated:** [February 3, 2022, 8:30pm UTC](https://discuss.elastic.co/t/database-connection/296167 "2022-02-03T20:30:00Z")
We want to connect Snowflake and PGSql Databases to Logstash/Elasticsearch. How do i go ahead with that? Is there any link to referral video or article? --- ## [Grok pattern for this specific log](https://discuss.elastic.co/t/grok-pattern-for-this-specific-log/296010)
**Author:** [@Mihailo\_Stanarevic](https://discuss.elastic.co/u/Mihailo_Stanarevic)\ **Replies:** 5\ **Last updated:** [February 3, 2022, 4:59pm UTC](https://discuss.elastic.co/t/grok-pattern-for-this-specific-log/296010 "2022-02-03T16:59:00Z")
I want to filter through this log: \[ ERROR\] 02.04.2016. 20:38:19 (FileManagerServlet:handleDownload) Date and time: Sat Apr 02 20:38:19 CEST 2016| miliseconds: 1459622299268| + session id: D4190DFF52C536C500FAF0947DB120… --- ## [Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id](https://discuss.elastic.co/t/failed-to-execute-action-action-logstash-create-pipeline-id/296138)
**Author:** [@jeromeat](https://discuss.elastic.co/u/jeromeat)\ **Replies:** 5\ **Last updated:** [February 3, 2022, 3:07pm UTC](https://discuss.elastic.co/t/failed-to-execute-action-action-logstash-create-pipeline-id/296138 "2022-02-03T15:07:51Z")
I'm currently trying to use Input http\_poller to make an API call to a site to pull some data. However I keep getting the following error: Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id… --- ## [Can the logstash rabbitmq input plugin read from the start of a stream?](https://discuss.elastic.co/t/can-the-logstash-rabbitmq-input-plugin-read-from-the-start-of-a-stream/296084)
**Author:** [@Daniel\_Bray](https://discuss.elastic.co/u/Daniel_Bray)\ **Replies:** 0\ **Last updated:** [February 2, 2022, 4:08pm UTC](https://discuss.elastic.co/t/can-the-logstash-rabbitmq-input-plugin-read-from-the-start-of-a-stream/296084 "2022-02-02T16:08:21Z")
I have a rabbitmq stream whose events I want logstash to process. I can get it working mostly ok but logstash only processes records that are pushed to the stream while logstash is running. If I push messages before lo… --- ## [Logstash configuration File](https://discuss.elastic.co/t/logstash-configuration-file/296179)
**Author:** [@harsh19](https://discuss.elastic.co/u/harsh19)\ **Replies:** 3\ **Last updated:** [February 3, 2022, 11:17am UTC](https://discuss.elastic.co/t/logstash-configuration-file/296179 "2022-02-03T11:17:10Z")
input { file { type =\> "json" path =\> "/home/harsh/Documents/elklog/2022/01/1/\*.json" start\_position =\> "beginning" sincedb\_path =\> "/dev/null" } } filter { } output { stdout { codec =\> rubydebug… --- ## [How to create grok filter to parse the below log format](https://discuss.elastic.co/t/how-to-create-grok-filter-to-parse-the-below-log-format/295988)
**Author:** [@poddraj](https://discuss.elastic.co/u/poddraj)\ **Replies:** 2\ **Last updated:** [February 3, 2022, 7:32am UTC](https://discuss.elastic.co/t/how-to-create-grok-filter-to-parse-the-below-log-format/295988 "2022-02-03T07:32:59Z")
2022-02-01 01:25:45.778-\[ActivationSrvc-28\] com.verizon.vnm.activation.ejb.helper.ActivationRequestHelper.nbaReqResponse(ActivationRequestHelper.java:3313) INFO {"VSAD\_ID":"ABFD","log\_type":"","app\_name":"ABC","VAST\_ID"… --- ## [Logstash Automation](https://discuss.elastic.co/t/logstash-automation/296150)
**Author:** [@harsh19](https://discuss.elastic.co/u/harsh19)\ **Replies:** 2\ **Last updated:** [February 3, 2022, 7:26am UTC](https://discuss.elastic.co/t/logstash-automation/296150 "2022-02-03T07:26:57Z")
there is a one Folder name logs in that Folder there is multiple json file like 1.json, 2.json, etc i want to pars logs in Elasticsearch with logstash so Question is that how to configure that all file in one conf fi… --- ## [Kafka input kibana message one field](https://discuss.elastic.co/t/kafka-input-kibana-message-one-field/296134)
**Author:** [@splitmessage88](https://discuss.elastic.co/u/splitmessage88)\ **Replies:** 2\ **Last updated:** [February 3, 2022, 7:00am UTC](https://discuss.elastic.co/t/kafka-input-kibana-message-one-field/296134 "2022-02-03T07:00:31Z")
Hello! I'm new here and have been stuck with an issue for 2 days. I have a working flow with winlogbeat - kafka - logstash - Elasticsearch. I receive the winlogbeat logs in kibana but the problem is that the full log … --- ## [Iteration In Logstash | Looping](https://discuss.elastic.co/t/iteration-in-logstash-looping/296148)
**Author:** [@Mayank\_Malhotra](https://discuss.elastic.co/u/Mayank_Malhotra)\ **Replies:** 1\ **Last updated:** [February 3, 2022, 6:59am UTC](https://discuss.elastic.co/t/iteration-in-logstash-looping/296148 "2022-02-03T06:59:52Z")
Hi All, I am creating a logstasg config in which I need to use iteration at some point giving an example below in terms of simpler programing language :- int n=300000; int counter=0,skip=0; while(counter\<300000) { skip… --- ## [Problem using fingerprint for my documents stored in elasticsearch](https://discuss.elastic.co/t/problem-using-fingerprint-for-my-documents-stored-in-elasticsearch/295101)
**Author:** [@MKH](https://discuss.elastic.co/u/MKH)\ **Replies:** 16\ **Last updated:** [February 2, 2022, 9:23pm UTC](https://discuss.elastic.co/t/problem-using-fingerprint-for-my-documents-stored-in-elasticsearch/295101 "2022-02-02T21:23:23Z")
Hi, I want to provide a unique id for the data I store in Elasticsearch index using fingerprint. But when I feed a bulk of docs to logstash I only one of the documents is stored in the index. I have 9 docs (9 json dicti… --- ## [Timestamp Event at Logstash Input](https://discuss.elastic.co/t/timestamp-event-at-logstash-input/296122)
**Author:** [@MakoWish](https://discuss.elastic.co/u/MakoWish)\ **Replies:** 0\ **Last updated:** [February 2, 2022, 8:24pm UTC](https://discuss.elastic.co/t/timestamp-event-at-logstash-input/296122 "2022-02-02T20:24:44Z")
We are using Persistent Queues, and I would like to calculate the time an event is in queue before actually being sent through the pipeline, but in order to do so, I need to add a timestamp in the input section of the pi… --- ## [Logstash reading from .log file is not consistent](https://discuss.elastic.co/t/logstash-reading-from-log-file-is-not-consistent/295661)
**Author:** [@HungNV](https://discuss.elastic.co/u/HungNV)\ **Replies:** 5\ **Last updated:** [February 2, 2022, 2:03pm UTC](https://discuss.elastic.co/t/logstash-reading-from-log-file-is-not-consistent/295661 "2022-02-02T14:03:21Z")
I got some error when logstash reading from .log file. Some time it cannot read whole content in a line. Im using logstash version 7.16.1 Here the input config: input { file { path =\> "${LOG\_PATH}" codec =\> json … --- ## [How to create grok/json filter to parse the below json format](https://discuss.elastic.co/t/how-to-create-grok-json-filter-to-parse-the-below-json-format/296022)
**Author:** [@Adabi\_Raihan](https://discuss.elastic.co/u/Adabi_Raihan)\ **Replies:** 9\ **Last updated:** [February 2, 2022, 8:49am UTC](https://discuss.elastic.co/t/how-to-create-grok-json-filter-to-parse-the-below-json-format/296022 "2022-02-02T08:49:28Z")
Hi Guys, I want to parse this JSON to Kibana using Logstash { "Format": "IDEA0", "ID": "2b03eb1f-fc4c-4f67-94e5-31c9fb32dccc", "DetectTime": "2022-01-31T08:16:12.600470+07:00", "EventTime": "2022-01-31T01:23:01.637438+… --- ## [Split fields in different docs](https://discuss.elastic.co/t/split-fields-in-different-docs/295895)
**Author:** [@cris](https://discuss.elastic.co/u/cris)\ **Replies:** 8\ **Last updated:** [February 2, 2022, 6:23am UTC](https://discuss.elastic.co/t/split-fields-in-different-docs/295895 "2022-02-02T06:23:54Z")
Hello friends. I am trying to reindex some data to another index but with a little differences. In the first index I have this kind of hit: { "France": { "Testing": { "status": "passed" } }, "Spain"… --- ## [Logstash is terminating with EACCES Permission denied](https://discuss.elastic.co/t/logstash-is-terminating-with-eacces-permission-denied/296032)
**Author:** [@HELIXInternational](https://discuss.elastic.co/u/HELIXInternational)\ **Replies:** 0\ **Last updated:** [February 2, 2022, 6:48am UTC](https://discuss.elastic.co/t/logstash-is-terminating-with-eacces-permission-denied/296032 "2022-02-02T06:48:00Z")
Hi All, logstash javapipeline is terminating Please can anyone help on this issue, It will be much helpful for us. logstash-plain \[2022-02-01T08:06:01,333\]\[INFO \]\[logstash.runner \] Log4j configuration path u… --- ## [Grok timeout for long messages](https://discuss.elastic.co/t/grok-timeout-for-long-messages/296009)
**Author:** [@Victorv18](https://discuss.elastic.co/u/Victorv18)\ **Replies:** 1\ **Last updated:** [February 1, 2022, 11:57pm UTC](https://discuss.elastic.co/t/grok-timeout-for-long-messages/296009 "2022-02-01T23:57:02Z")
I have a grok filter parsing logs from a DB2 AIX server, and the logs are sent to Logstash in the .txt format, the file contains a lot of log messages, separated by a timestamp parameter And i receive a timeout message … --- ## [Config logstash with elasticsearch cloud](https://discuss.elastic.co/t/config-logstash-with-elasticsearch-cloud/295962)
**Author:** [@Cheroufa](https://discuss.elastic.co/u/Cheroufa)\ **Replies:** 1\ **Last updated:** [February 1, 2022, 8:34pm UTC](https://discuss.elastic.co/t/config-logstash-with-elasticsearch-cloud/295962 "2022-02-01T20:34:54Z")
hi guys i want to configure logstash with elasticsearch cloud to use the Centralized Pipeline Management but logstash can't reach the cloud elasticsearch cluster. logs : \[2022-02-02T16:19:25,986\]\[ERROR\]\[logstash.lice… --- ## [Rollover ILM policy for existing Index](https://discuss.elastic.co/t/rollover-ilm-policy-for-existing-index/295885)
**Author:** [@Vani1](https://discuss.elastic.co/u/Vani1)\ **Replies:** 5\ **Last updated:** [February 1, 2022, 6:39pm UTC](https://discuss.elastic.co/t/rollover-ilm-policy-for-existing-index/295885 "2022-02-01T18:39:59Z")
Hi Can anyone help me on Rollover ILM for existing index , which needs to be rolled over on reaching certain age/ gb. Thankyou in advance. --- ## [Splitting the log into a csv file](https://discuss.elastic.co/t/splitting-the-log-into-a-csv-file/295922)
**Author:** [@PanPiotr](https://discuss.elastic.co/u/PanPiotr)\ **Replies:** 1\ **Last updated:** [February 1, 2022, 6:08pm UTC](https://discuss.elastic.co/t/splitting-the-log-into-a-csv-file/295922 "2022-02-01T18:08:31Z")
Hi, I want to use logstash to separate the appropriate logs by a constant value appearing in these logs, and then divide the log into pieces after the separator ("|") and put it into a csv file with headers. The logs I'… --- ## [Unable to change date format](https://discuss.elastic.co/t/unable-to-change-date-format/295948)
**Author:** [@duanra22](https://discuss.elastic.co/u/duanra22)\ **Replies:** 3\ **Last updated:** [February 1, 2022, 3:05pm UTC](https://discuss.elastic.co/t/unable-to-change-date-format/295948 "2022-02-01T15:05:05Z")
Hello everyone, I've seen several topics with the same problem, however none of the solutions I found worked for me and it's making me crazy because I am sure it should be very simple. To explain the problem, I am usin… --- ## [Keystore for logstash](https://discuss.elastic.co/t/keystore-for-logstash/295688)
**Author:** [@Cheroufa](https://discuss.elastic.co/u/Cheroufa)\ **Replies:** 3\ **Last updated:** [February 1, 2022, 12:18pm UTC](https://discuss.elastic.co/t/keystore-for-logstash/295688 "2022-02-01T12:18:51Z")
Hi guys i installed a keystore for logstash 7.16.3 in local on my machine it works fine but when i deploy the same pipeline on kibana =\> logstash pipelines it does'nt work. log : "Unable to configure plugins: (Configu… --- ## [Logstash error during configuration](https://discuss.elastic.co/t/logstash-error-during-configuration/295911)
**Author:** [@harsh19](https://discuss.elastic.co/u/harsh19)\ **Replies:** 4\ **Last updated:** [February 1, 2022, 9:22am UTC](https://discuss.elastic.co/t/logstash-error-during-configuration/295911 "2022-02-01T09:22:19Z")
When i Run ./logstash -f logstash-simple.conf then it's give me an error like this, Using bundled JDK: /usr/share/logstash/jdk OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 a… --- ## [Logstash is merging messages sent through http](https://discuss.elastic.co/t/logstash-is-merging-messages-sent-through-http/295905)
**Author:** [@Ruben\_Bracamonte](https://discuss.elastic.co/u/Ruben_Bracamonte)\ **Replies:** 0\ **Last updated:** [February 1, 2022, 6:51am UTC](https://discuss.elastic.co/t/logstash-is-merging-messages-sent-through-http/295905 "2022-02-01T06:51:42Z")
Hi, I have a basic Logstash config with HTTP as input. My API is writing logs every 2-3 seconds, and sometimes on Kibana I can see two messages, with their two timestamps and two different IPs (from the caller), etc in … --- ## [Azure Event Hub, The client could not finish the operation within specified maximum execution timeout](https://discuss.elastic.co/t/azure-event-hub-the-client-could-not-finish-the-operation-within-specified-maximum-execution-timeout/295875)
**Author:** [@ElasticLiver](https://discuss.elastic.co/u/ElasticLiver)\ **Replies:** 0\ **Last updated:** [January 31, 2022, 8:34pm UTC](https://discuss.elastic.co/t/azure-event-hub-the-client-could-not-finish-the-operation-within-specified-maximum-execution-timeout/295875 "2022-01-31T20:34:31Z")
Hi Im having this Warns when I try to get events from event hub. com.microsoft.azure.storage.StorageException: The client could not finish the operation within specified maximum execution timeout. after those warns I g… --- ## [Finding bottleneck in pipeline](https://discuss.elastic.co/t/finding-bottleneck-in-pipeline/294308)
**Author:** [@YvorL](https://discuss.elastic.co/u/YvorL)\ **Replies:** 8\ **Last updated:** [February 1, 2022, 1:19am UTC](https://discuss.elastic.co/t/finding-bottleneck-in-pipeline/294308 "2022-02-01T01:19:51Z")
Hello, I'm looking for brave and clever people who can help out someone in a pinch. I've been struggling to find out why one of my (very busy) index is lagging behind. I've been reading different posts, suggestions, et… --- ## [Elasticsearch 7.10.2 Compatibility with Logstash 7.16.3](https://discuss.elastic.co/t/elasticsearch-7-10-2-compatibility-with-logstash-7-16-3/295889)
**Author:** [@andreakatie](https://discuss.elastic.co/u/andreakatie)\ **Replies:** 2\ **Last updated:** [January 31, 2022, 11:56pm UTC](https://discuss.elastic.co/t/elasticsearch-7-10-2-compatibility-with-logstash-7-16-3/295889 "2022-01-31T23:56:36Z")
I originally tried to upgrade Elasticsearch to 7.10.2 with the OpenDistro 1.13.1 update, but it keeps giving me an error (shown below). The compatibility matrix shows that they should work together. If anyone knows wheth… --- ## [Http input plugin](https://discuss.elastic.co/t/http-input-plugin/295609)
**Author:** [@steevhise](https://discuss.elastic.co/u/steevhise)\ **Replies:** 3\ **Last updated:** [January 31, 2022, 10:39pm UTC](https://discuss.elastic.co/t/http-input-plugin/295609 "2022-01-31T22:39:47Z")
I'm trying to debug communication with an endpoint set up in Logstash using the http input plugin. I'd like to be able to see in the logstash log the raw request payload that logstash receives but I can't figure out how … --- ## [Merge data and metadata csv file in logstash](https://discuss.elastic.co/t/merge-data-and-metadata-csv-file-in-logstash/295815)
**Author:** [@danova](https://discuss.elastic.co/u/danova)\ **Replies:** 2\ **Last updated:** [January 31, 2022, 8:13pm UTC](https://discuss.elastic.co/t/merge-data-and-metadata-csv-file-in-logstash/295815 "2022-01-31T20:13:05Z")
Hi, I´m trying to import/combine two csv files as follows logstash\_metadata.csv: Sample,Treatment,Code S4444003,T\_7896\_D3,G10 S4444004,T\_4516\_D0t1h,G01 logstash\_file.csv: Sample,genus,value S4444003,Chloronema… --- ## [Logstash index template + data streams](https://discuss.elastic.co/t/logstash-index-template-data-streams/295872)
**Author:** [@cmanzur](https://discuss.elastic.co/u/cmanzur)\ **Replies:** 0\ **Last updated:** [January 31, 2022, 7:39pm UTC](https://discuss.elastic.co/t/logstash-index-template-data-streams/295872 "2022-01-31T19:39:25Z")
I'm trying to use data streams and index templates in logstash v7.17 What is the right Elasticsearch output configuration to achieve this? Option 1: Using data stream in the template Fails output { elasticse… --- ## [Only String and Array types are splittable. field:\[array\]\[method\] is of type = NilClass"](https://discuss.elastic.co/t/only-string-and-array-types-are-splittable-field-array-method-is-of-type-nilclass/295861)
**Author:** [@rutika\_kamble](https://discuss.elastic.co/u/rutika_kamble)\ **Replies:** 0\ **Last updated:** [January 31, 2022, 3:21pm UTC](https://discuss.elastic.co/t/only-string-and-array-types-are-splittable-field-array-method-is-of-type-nilclass/295861 "2022-01-31T15:21:39Z")
I am trying to input JSON data from logs through logstash to Elasticsearch but I am getting this error field:\[array\]\[method\] is of type = NilClass" Preformatted text Please find my logstash conf file- input { http { … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=162) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=164)