# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=267 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 268 --- ## [Can't parse Unix timestamp with logstash. Getting \_dateparsefailure](https://discuss.elastic.co/t/cant-parse-unix-timestamp-with-logstash-getting-dateparsefailure/259718)
**Author:** [@Webtrend\_Inc](https://discuss.elastic.co/u/Webtrend_Inc)\ **Replies:** 4\ **Last updated:** [January 4, 2021, 2:05am UTC](https://discuss.elastic.co/t/cant-parse-unix-timestamp-with-logstash-getting-dateparsefailure/259718 "2021-01-04T02:05:09Z")
Hi, Our log files are in json format. Here is a small snippet of the file { "timestamp": 1608191939682, "formatVersion": 1, "webaclId": "69c2a78d-d849-4dca-bccd-xxx", "terminatingRuleId": "Default\_Action", "te… --- ## [Parsing json array with logstash](https://discuss.elastic.co/t/parsing-json-array-with-logstash/259852)
**Author:** [@Webtrend\_Inc](https://discuss.elastic.co/u/Webtrend_Inc)\ **Replies:** 3\ **Last updated:** [January 4, 2021, 2:04am UTC](https://discuss.elastic.co/t/parsing-json-array-with-logstash/259852 "2021-01-04T02:04:01Z")
Our logs have http headers as a json array as shown below "httpRequest": { "httpVersion": "HTTP/1.1", "args": "", "country": "US", "requestId": "1-5fec0c53-78cbd10c4248df7666a62d78", "clien… --- ## [Problem with executing path to Logstash.bat](https://discuss.elastic.co/t/problem-with-executing-path-to-logstash-bat/259731)
**Author:** [@keyofstarrysky973](https://discuss.elastic.co/u/keyofstarrysky973)\ **Replies:** 2\ **Last updated:** [January 3, 2021, 7:19pm UTC](https://discuss.elastic.co/t/problem-with-executing-path-to-logstash-bat/259731 "2021-01-03T19:19:33Z")
Hello Everyone, Pls I need some help on this problem I'm having executing Logstash on Windows 10. I have an ELK Stack with both Elasticsearch and Kibana working without any problems. I have a directory, named elk\_stack … --- ## [Issue installing Logstash no such file to load -- bundler](https://discuss.elastic.co/t/issue-installing-logstash-no-such-file-to-load-bundler/260041)
**Author:** [@Grant1999](https://discuss.elastic.co/u/Grant1999)\ **Replies:** 5\ **Last updated:** [January 3, 2021, 6:15pm UTC](https://discuss.elastic.co/t/issue-installing-logstash-no-such-file-to-load-bundler/260041 "2021-01-03T18:15:29Z")
I have downloaded kibana and elastic search on windows 10 64 bit and now trying to install logstash. But I get this error when I type in the command any ideas? C:\\Elasticstack\\logstash-7.10.1\>logstash -f logstash.conf U… --- ## [How to test logstash .conf file easily without logstash shutting down everytime?](https://discuss.elastic.co/t/how-to-test-logstash-conf-file-easily-without-logstash-shutting-down-everytime/260004)
**Author:** [@kelk](https://discuss.elastic.co/u/kelk)\ **Replies:** 3\ **Last updated:** [January 1, 2021, 10:10pm UTC](https://discuss.elastic.co/t/how-to-test-logstash-conf-file-easily-without-logstash-shutting-down-everytime/260004 "2021-01-01T22:10:29Z")
I'm currently testing the logstash in below manner. /opt/logstash/bin/logstash -f /tmp/simpleInputs.conf This works perfectly, but the issue is, it takes some time to execute it.. just for the starting of java process … --- ## [Reading data trough mongodb -input](https://discuss.elastic.co/t/reading-data-trough-mongodb-input/259999)
**Author:** [@bucco1958](https://discuss.elastic.co/u/bucco1958)\ **Replies:** 1\ **Last updated:** [January 1, 2021, 5:39pm UTC](https://discuss.elastic.co/t/reading-data-trough-mongodb-input/259999 "2021-01-01T17:39:26Z")
I'm trying to learn more about the mongodb-input-plugin. I'm running a local mongo db instance on my localhost on my Win 10 machine. The database is up and used with other purposes. There is no security on the database -… --- ## [Can't access inner fields in JSON](https://discuss.elastic.co/t/cant-access-inner-fields-in-json/259978)
**Author:** [@ake](https://discuss.elastic.co/u/ake)\ **Replies:** 6\ **Last updated:** [December 31, 2020, 10:34pm UTC](https://discuss.elastic.co/t/cant-access-inner-fields-in-json/259978 "2020-12-31T22:34:50Z")
Hi everyone, I'm trying to extract some fields from JSON that I'm getting from MySQL and add it to ES. I've been going over way too many discussions here and some other sites but can't find the solution for my problem. … --- ## [Logstash HA](https://discuss.elastic.co/t/logstash-ha/258979)
**Author:** [@ksaha](https://discuss.elastic.co/u/ksaha)\ **Replies:** 5\ **Last updated:** [December 31, 2020, 3:51pm UTC](https://discuss.elastic.co/t/logstash-ha/258979 "2020-12-31T15:51:20Z")
Hi, Can anyone please provide what are the various options of implementing Logstash High Availability? I am aware of using at least two Logstash instance for Logstash HA. Is there any other way by which I can achieve L… --- ## [Logstash execption in pipeline worker](https://discuss.elastic.co/t/logstash-execption-in-pipeline-worker/259965)
**Author:** [@Mohammad\_Mousavi](https://discuss.elastic.co/u/Mohammad_Mousavi)\ **Replies:** 1\ **Last updated:** [December 31, 2020, 3:17pm UTC](https://discuss.elastic.co/t/logstash-execption-in-pipeline-worker/259965 "2020-12-31T15:17:26Z")
Hi, I have 4 logstash ( v7.4.2) services on 4 separate VMs. The logstash services keeps crashing after a while or few hours. This is the config: filter { ip2location { source =\> "clientip" … --- ## [\[Logstash monitoring with metricbeat\]: http server gave http response to HTTPs client](https://discuss.elastic.co/t/logstash-monitoring-with-metricbeat-http-server-gave-http-response-to-https-client/259950)
**Author:** [@Abdelhalim](https://discuss.elastic.co/u/Abdelhalim)\ **Replies:** 0\ **Last updated:** [December 31, 2020, 10:22am UTC](https://discuss.elastic.co/t/logstash-monitoring-with-metricbeat-http-server-gave-http-response-to-https-client/259950 "2020-12-31T10:22:13Z")
Hello, I am trying to monitor my cluster using metricbeat, for elasticsearch and kibana nodes it's working perfectly, but when I try to monitor logstash I am getting these errors from my metricbeat logs 2020-12-31T11:0… --- ## [Elasticsearch template in Logstash doesn't mapping and not able to sort fields](https://discuss.elastic.co/t/elasticsearch-template-in-logstash-doesnt-mapping-and-not-able-to-sort-fields/259937)
**Author:** [@rknd](https://discuss.elastic.co/u/rknd)\ **Replies:** 0\ **Last updated:** [December 31, 2020, 7:30am UTC](https://discuss.elastic.co/t/elasticsearch-template-in-logstash-doesnt-mapping-and-not-able-to-sort-fields/259937 "2020-12-31T07:30:18Z")
I want to sort datas via elasticsearch rest client, I want to get data that match with the name "match\_field", after this match sort via "int\_var", below is my template in logstash. { "index\_patterns": \["index\_name"\]… --- ## [Facing issue with Logstash 7.6.2 \[logstash.outputs.elasticsearch\]\[.monitoring-logstash\] Encountered a retryable error](https://discuss.elastic.co/t/facing-issue-with-logstash-7-6-2-logstash-outputs-elasticsearch-monitoring-logstash-encountered-a-retryable-error/259923)
**Author:** [@Usman18](https://discuss.elastic.co/u/Usman18)\ **Replies:** 1\ **Last updated:** [December 30, 2020, 10:23pm UTC](https://discuss.elastic.co/t/facing-issue-with-logstash-7-6-2-logstash-outputs-elasticsearch-monitoring-logstash-encountered-a-retryable-error/259923 "2020-12-30T22:23:43Z")
I have upgraded logstash and now I am using logstash version 7.6.2. My elasticsearch version is 6.8.13. After upgrade, I see a lot of error logs related to monitoring-logstash pipeline. The log of this error is given bel… --- ## [Scaling logstash nodes](https://discuss.elastic.co/t/scaling-logstash-nodes/259792)
**Author:** [@David\_Beradze](https://discuss.elastic.co/u/David_Beradze)\ **Replies:** 6\ **Last updated:** [December 30, 2020, 7:43pm UTC](https://discuss.elastic.co/t/scaling-logstash-nodes/259792 "2020-12-30T19:43:11Z")
Hello, We have oracle (single table) \>\> logstash \>\> elasticsearch. What is the way to scale horizontally logstash nodes to prevent the same data selection from the same source? (oracle table) --- ## [Read actual and previous line using plugin file in Logstash](https://discuss.elastic.co/t/read-actual-and-previous-line-using-plugin-file-in-logstash/259823)
**Author:** [@dooger21](https://discuss.elastic.co/u/dooger21)\ **Replies:** 2\ **Last updated:** [December 30, 2020, 4:36pm UTC](https://discuss.elastic.co/t/read-actual-and-previous-line-using-plugin-file-in-logstash/259823 "2020-12-30T16:36:31Z")
Hello everyone, I need to read a log in tail mode and for each record read in real time to also obtain the previous record. That is, whatever you read the row (11) that has the date 20200607-00:10:00 also get the previo… --- ## [Multiple output options in logstash.conf](https://discuss.elastic.co/t/multiple-output-options-in-logstash-conf/259877)
**Author:** [@nyquillus](https://discuss.elastic.co/u/nyquillus)\ **Replies:** 8\ **Last updated:** [December 30, 2020, 1:45pm UTC](https://discuss.elastic.co/t/multiple-output-options-in-logstash-conf/259877 "2020-12-30T13:45:54Z")
Hi, I'm trying to come up with a logstash.conf to seperate the indexing method between 2 different sources. One will be daily index and other will be monthly index. I came up with something like this but it didn't work. … --- ## [Logstash S3 Input Plugin not recognizing IAM Instance Profile](https://discuss.elastic.co/t/logstash-s3-input-plugin-not-recognizing-iam-instance-profile/259812)
**Author:** [@prod](https://discuss.elastic.co/u/prod)\ **Replies:** 2\ **Last updated:** [December 30, 2020, 12:47pm UTC](https://discuss.elastic.co/t/logstash-s3-input-plugin-not-recognizing-iam-instance-profile/259812 "2020-12-30T12:47:44Z")
Checking the S3 Input plugin documentation This plugin uses the AWS SDK and supports several ways to get credentials, which will be tried in this order: Static configuration, using access\_key\_id and secret\_access\_k… --- ## [Logstash grok field not found](https://discuss.elastic.co/t/logstash-grok-field-not-found/259864)
**Author:** [@abu.sayeed](https://discuss.elastic.co/u/abu.sayeed)\ **Replies:** 0\ **Last updated:** [December 30, 2020, 9:10am UTC](https://discuss.elastic.co/t/logstash-grok-field-not-found/259864 "2020-12-30T09:10:11Z")
Thanks all --- ## [Logstask nested field problem](https://discuss.elastic.co/t/logstask-nested-field-problem/259781)
**Author:** [@abu.sayeed](https://discuss.elastic.co/u/abu.sayeed)\ **Replies:** 2\ **Last updated:** [December 30, 2020, 10:41am UTC](https://discuss.elastic.co/t/logstask-nested-field-problem/259781 "2020-12-30T10:41:10Z")
Logs message : "message" =\> "Backoff FixedBackOff{interval=0, currentAttempts=10, maxAttempts=9} exhausted " I need field like below: "FixedBackOff" =\> { "interval"=0, "currentAttempts"=10, "maxAttempts"=9 } Than… --- ## [Date filter plugin not working as expected](https://discuss.elastic.co/t/date-filter-plugin-not-working-as-expected/259835)
**Author:** [@S3l3ct3d](https://discuss.elastic.co/u/S3l3ct3d)\ **Replies:** 8\ **Last updated:** [December 29, 2020, 11:48pm UTC](https://discuss.elastic.co/t/date-filter-plugin-not-working-as-expected/259835 "2020-12-29T23:48:33Z")
Good evening, I have a rather extensive .conf file with a Date filter plugin being used. However it is not actually working. I am sending it to a target of @timestamp, but, I am not seeing the corresponding Original Eve… --- ## [Logstash s3 input plugin not deleting files after processing](https://discuss.elastic.co/t/logstash-s3-input-plugin-not-deleting-files-after-processing/259302)
**Author:** [@mcjay](https://discuss.elastic.co/u/mcjay)\ **Replies:** 1\ **Last updated:** [December 29, 2020, 7:48pm UTC](https://discuss.elastic.co/t/logstash-s3-input-plugin-not-deleting-files-after-processing/259302 "2020-12-29T19:48:14Z")
I am using s3 input plugin on logstash to read AWS VPC flow logs from s3 bucket. I have this as my input.conf file input { s3 { bucket =\> "vpc-flow-logs-elk" prefix =\> "AWSLogs/200312345677/vpcflowlogs/us-east-1/" regio… --- ## [Logstash re-read full log after restart](https://discuss.elastic.co/t/logstash-re-read-full-log-after-restart/259796)
**Author:** [@msk\_76](https://discuss.elastic.co/u/msk_76)\ **Replies:** 5\ **Last updated:** [December 29, 2020, 5:33pm UTC](https://discuss.elastic.co/t/logstash-re-read-full-log-after-restart/259796 "2020-12-29T17:33:03Z")
I am facing an issue with logstash that it read the log from begin even after it has loaded the log file already. It happens only when i restart the logstash. Inside the .sincedb file the contents are : 1149101767 0 38 … --- ## [Hello, Help for Error : An unexpected error occurred! {:error=\>#\ **Author:** [@Muhammad\_Firdaus](https://discuss.elastic.co/u/Muhammad_Firdaus)\ **Replies:** 1\ **Last updated:** [December 29, 2020, 3:11pm UTC](https://discuss.elastic.co/t/hello-help-for-error-an-unexpected-error-occurred-error-logstash-dont-know-how-to-handle/259787 "2020-12-29T15:11:51Z") Get Error when starting up Logstash \[2020-12-29T16:08:40,925\]\[ERROR\]\[logstash.agent \] An exception happened when converging configuration {:exception=\>LogStash::Error, :message=\>"Don't know how to handle Java:… --- ## [Dynamic urls for Logstash http\_poller plugin](https://discuss.elastic.co/t/dynamic-urls-for-logstash-http-poller-plugin/259785)
**Author:** [@malsioufi](https://discuss.elastic.co/u/malsioufi)\ **Replies:** 7\ **Last updated:** [December 29, 2020, 2:57pm UTC](https://discuss.elastic.co/t/dynamic-urls-for-logstash-http-poller-plugin/259785 "2020-12-29T14:57:05Z")
Hi, I have an implementation of some script that calls an api to start some job, I can follow up the job status another api (jobs-api). I want to use Logstash and Elasticsearch to monitor the status of the jobs I start, … --- ## [In filebeat want to enable system module, postgressql modules, application logs and how i can write logstash pipeline](https://discuss.elastic.co/t/in-filebeat-want-to-enable-system-module-postgressql-modules-application-logs-and-how-i-can-write-logstash-pipeline/259772)
**Author:** [@prakash22](https://discuss.elastic.co/u/prakash22)\ **Replies:** 1\ **Last updated:** [December 29, 2020, 6:03am UTC](https://discuss.elastic.co/t/in-filebeat-want-to-enable-system-module-postgressql-modules-application-logs-and-how-i-can-write-logstash-pipeline/259772 "2020-12-29T06:03:41Z")
In filebeat want to enable system module, postgressql modules, application logs and how i can write logstash pipeline --- ## [Insert Custom Field Name into Index Name after going through Solace Pubsub](https://discuss.elastic.co/t/insert-custom-field-name-into-index-name-after-going-through-solace-pubsub/259639)
**Author:** [@ckough](https://discuss.elastic.co/u/ckough)\ **Replies:** 1\ **Last updated:** [December 29, 2020, 1:31am UTC](https://discuss.elastic.co/t/insert-custom-field-name-into-index-name-after-going-through-solace-pubsub/259639 "2020-12-29T01:31:39Z")
Hi there I'm using ELK 7.10. My test setup is as follows: filebeat (running in kibana) -\> logstash01 -\> solace pubsub+ -\> logstash02 -\> Elasticsearch -\> kibana What I'm trying to accomplish: Insert custom fields cre… --- ## [Unable to create a new index pattern / indexes not loading](https://discuss.elastic.co/t/unable-to-create-a-new-index-pattern-indexes-not-loading/259527)
**Author:** [@elk\_guy\_1234](https://discuss.elastic.co/u/elk_guy_1234)\ **Replies:** 19\ **Last updated:** [December 28, 2020, 4:50pm UTC](https://discuss.elastic.co/t/unable-to-create-a-new-index-pattern-indexes-not-loading/259527 "2020-12-28T16:50:57Z")
Problem: When trying to create a new index pattern the error "The index pattern you've entered does not match any indices". Really the problem is my index's will not load from a config / schema. Index Details: This inde… --- ## [How to keep logstash date type?](https://discuss.elastic.co/t/how-to-keep-logstash-date-type/259742)
**Author:** [@pamiers](https://discuss.elastic.co/u/pamiers)\ **Replies:** 2\ **Last updated:** [December 28, 2020, 4:39pm UTC](https://discuss.elastic.co/t/how-to-keep-logstash-date-type/259742 "2020-12-28T16:39:36Z")
I am reading data from oracle and putting data into elasticsearch through logstash. However, when reading data from oracle, the value of the field corresponding to the Date type is automatically changed to UTC type, dif… --- ## [Logstash enrichement: URLHaus](https://discuss.elastic.co/t/logstash-enrichement-urlhaus/259720)
**Author:** [@TheHunter1](https://discuss.elastic.co/u/TheHunter1)\ **Replies:** 5\ **Last updated:** [December 28, 2020, 4:14pm UTC](https://discuss.elastic.co/t/logstash-enrichement-urlhaus/259720 "2020-12-28T16:14:18Z")
Hello , I would like to enrich my SIEM with information from URLhaus, I configured my logstash pipline like this: input { exec { command =\> 'curl https://urlhaus.abuse.ch/downloads/csv/' interval =\> 86400 … --- ## [Logstash grok hour and minute ruby](https://discuss.elastic.co/t/logstash-grok-hour-and-minute-ruby/259724)
**Author:** [@Cristiane\_Marcarini](https://discuss.elastic.co/u/Cristiane_Marcarini)\ **Replies:** 0\ **Last updated:** [December 28, 2020, 11:16am UTC](https://discuss.elastic.co/t/logstash-grok-hour-and-minute-ruby/259724 "2020-12-28T11:16:48Z")
Hi, I have a grok and I don't want to change it, but I wanted to have the hour and minutes separate, it could be using Ruby, something like that. Thanks else if "performanceLog" in \[path\] { grok { match =\> {"mess… --- ## [How to save only one data into mongodb by logstash-output-mongodb plugin?](https://discuss.elastic.co/t/how-to-save-only-one-data-into-mongodb-by-logstash-output-mongodb-plugin/259702)
**Author:** [@iooi](https://discuss.elastic.co/u/iooi)\ **Replies:** 0\ **Last updated:** [December 28, 2020, 3:58am UTC](https://discuss.elastic.co/t/how-to-save-only-one-data-into-mongodb-by-logstash-output-mongodb-plugin/259702 "2020-12-28T03:58:22Z")
Like this output configuration, just want to send id to mongodb output { mongodb { uri =\> 'mongodb://localhost' database =\> "testDB" collection =\> 'testCollection' codec =\> line { format =\> "%{id}"} … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=266) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=268)