# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=175 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 176 --- ## [Logstash Error: "Your settings are invalid"](https://discuss.elastic.co/t/logstash-error-your-settings-are-invalid/291544)
**Author:** [@neerajg](https://discuss.elastic.co/u/neerajg)\ **Replies:** 2\ **Last updated:** [December 13, 2021, 7:52am UTC](https://discuss.elastic.co/t/logstash-error-your-settings-are-invalid/291544 "2021-12-13T07:52:03Z")
Hi guys, I just upgraded elk stack from 7.15.0 to 7.16.0. Everything is working fine except Logstash. It ends up failing to start. I checked the system logs and found the following error. Error: logstash\[20212\]: Ope… --- ## [Output to different elastic search Indices based on the string](https://discuss.elastic.co/t/output-to-different-elastic-search-indices-based-on-the-string/291568)
**Author:** [@mussa572](https://discuss.elastic.co/u/mussa572)\ **Replies:** 3\ **Last updated:** [December 13, 2021, 3:40am UTC](https://discuss.elastic.co/t/output-to-different-elastic-search-indices-based-on-the-string/291568 "2021-12-13T03:40:17Z")
Hi I have a Category field (cat) which can have following value and future expect to add more. BL LA KA Currently I have following code to output to the different Elasticsearch indices from our Logstash pipeline . I… --- ## [Logstash ingest pipeline - Data from one pipeline going to another](https://discuss.elastic.co/t/logstash-ingest-pipeline-data-from-one-pipeline-going-to-another/289131)
**Author:** [@parthmaniar](https://discuss.elastic.co/u/parthmaniar)\ **Replies:** 8\ **Last updated:** [December 12, 2021, 9:35am UTC](https://discuss.elastic.co/t/logstash-ingest-pipeline-data-from-one-pipeline-going-to-another/289131 "2021-12-12T09:35:42Z")
Hello, I hope my message finds the members of the community and their loved ones safe and healthy. I am running logstash (7.15.2) on a Raspberry Pi 4B running Ubuntu 20.04.3 LTS. I have seven pipelines, all listening … --- ## [Split index in runtime](https://discuss.elastic.co/t/split-index-in-runtime/291329)
**Author:** [@valentineg](https://discuss.elastic.co/u/valentineg)\ **Replies:** 2\ **Last updated:** [December 12, 2021, 8:01am UTC](https://discuss.elastic.co/t/split-index-in-runtime/291329 "2021-12-12T08:01:11Z")
HI, we have logstash and elastic on a k8s environment. in our topology logstash receives data from multiple applications while part of the data is application name \\ id. is there a way to add a dynamic variable to the … --- ## [Index roll over every 30 minutes](https://discuss.elastic.co/t/index-roll-over-every-30-minutes/290788)
**Author:** [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)\ **Replies:** 10\ **Last updated:** [December 11, 2021, 5:35pm UTC](https://discuss.elastic.co/t/index-roll-over-every-30-minutes/290788 "2021-12-11T17:35:24Z")
Hello All, We have a logstash set up where input is based on a file. Logstash parses the file and an index is created. A new file ends up for processing every 30 minutes. We want a new index to be created every time a… --- ## [MySQL Query Equivalent In JDBC Static](https://discuss.elastic.co/t/mysql-query-equivalent-in-jdbc-static/290347)
**Author:** [@rojin](https://discuss.elastic.co/u/rojin)\ **Replies:** 3\ **Last updated:** [December 11, 2021, 6:28am UTC](https://discuss.elastic.co/t/mysql-query-equivalent-in-jdbc-static/290347 "2021-12-11T06:28:42Z")
hello, I have a query in MySQL and I was wondering what would the equal query in JDBC static with parameters be. Here's my query: select name from my\_table where inet\_aton('1.1.1.1') between ip\_from and ip\_to limit 1;… --- ## [GROK Filter - Extract Year and Month](https://discuss.elastic.co/t/grok-filter-extract-year-and-month/291470)
**Author:** [@JeremyP](https://discuss.elastic.co/u/JeremyP)\ **Replies:** 1\ **Last updated:** [December 10, 2021, 10:01pm UTC](https://discuss.elastic.co/t/grok-filter-extract-year-and-month/291470 "2021-12-10T22:01:45Z")
Hi.... I'm looking for some help on creating a new field based on a grok pattern. Here is the field in question: "last\_assessed\_for\_vulnerabilities": "2021-12-10T07:05:41.154Z" I'm looking to create the following fiel… --- ## [Error installing logstash codec plugin](https://discuss.elastic.co/t/error-installing-logstash-codec-plugin/291472)
**Author:** [@lucky7s](https://discuss.elastic.co/u/lucky7s)\ **Replies:** 0\ **Last updated:** [December 10, 2021, 9:16pm UTC](https://discuss.elastic.co/t/error-installing-logstash-codec-plugin/291472 "2021-12-10T21:16:59Z")
I downloaded the logstash-codec-leef plugin from github and successfully ran 'build install' using jruby. All rspec tests succeed. When I try to install this plugin using 'sudo logstash-plugin install' I get the followi… --- ## [Problems with Logstash Pipeline to ElasticSearch](https://discuss.elastic.co/t/problems-with-logstash-pipeline-to-elasticsearch/290936)
**Author:** [@Leandro\_Valim](https://discuss.elastic.co/u/Leandro_Valim)\ **Replies:** 12\ **Last updated:** [December 10, 2021, 7:51pm UTC](https://discuss.elastic.co/t/problems-with-logstash-pipeline-to-elasticsearch/290936 "2021-12-10T19:51:48Z")
Hi i have some erros to up my logstash service as bellows \[2021-12-03T19:44:46,535\]\[INFO \]\[logstash.javapipeline \]\[KSC\] Pipeline terminated {"pipeline.id"=\>"KSC"} \[2021-12-03T19:44:46,550\]\[ERROR\]\[logstash.agent … --- ## [Configure logstash to receive different csv files with different fields](https://discuss.elastic.co/t/configure-logstash-to-receive-different-csv-files-with-different-fields/291451)
**Author:** [@sanchez](https://discuss.elastic.co/u/sanchez)\ **Replies:** 1\ **Last updated:** [December 10, 2021, 6:16pm UTC](https://discuss.elastic.co/t/configure-logstash-to-receive-different-csv-files-with-different-fields/291451 "2021-12-10T18:16:08Z")
Hello good mornig for everyone I'm a beginner at Elasticsearch, Kibana, and Logstash I am trying to configure logstash to receive different csv files with different fields, how can I configure logstash to be able to de… --- ## [Could not index event to Elasticsearch](https://discuss.elastic.co/t/could-not-index-event-to-elasticsearch/291397)
**Author:** [@bropid](https://discuss.elastic.co/u/bropid)\ **Replies:** 2\ **Last updated:** [December 10, 2021, 4:53pm UTC](https://discuss.elastic.co/t/could-not-index-event-to-elasticsearch/291397 "2021-12-10T16:53:15Z")
Hi, i'm try to parse the ngfw event with logstash in tcp port 5000 to Elasticsearch but the logstash-plain.log always said: "status"=\>400, "error"=\>{"type"=\>"ma … --- ## [Encountered a retryable error (will retry with exponential backoff) {:code=\>413,](https://discuss.elastic.co/t/encountered-a-retryable-error-will-retry-with-exponential-backoff-code-413/291417)
**Author:** [@danushkalakmina](https://discuss.elastic.co/u/danushkalakmina)\ **Replies:** 1\ **Last updated:** [December 10, 2021, 2:36pm UTC](https://discuss.elastic.co/t/encountered-a-retryable-error-will-retry-with-exponential-backoff-code-413/291417 "2021-12-10T14:36:20Z")
Hi there i got this error in my logstash while sending logs to the Elasticsearch cluster.When i restarted the logstash service it got fix. But after 1-2 days it will happen recursively. \[2021-12-10T10:17:52,948\]\[ERROR\]… --- ## [Logs not being sent to logstash from .net application](https://discuss.elastic.co/t/logs-not-being-sent-to-logstash-from-net-application/291401)
**Author:** [@Murad000](https://discuss.elastic.co/u/Murad000)\ **Replies:** 1\ **Last updated:** [December 10, 2021, 2:31pm UTC](https://discuss.elastic.co/t/logs-not-being-sent-to-logstash-from-net-application/291401 "2021-12-10T14:31:48Z")
Hello, I launched elk stack on docker but logstash isn't receiving any logs from my application. I cloned this repo,added RUN logstash-plugin install logstash-input-http to the logstash's Dockerfile and changed the con… --- ## [Prune filter - Whitelist JSON subfields & per-pipeline log file](https://discuss.elastic.co/t/prune-filter-whitelist-json-subfields-per-pipeline-log-file/291298)
**Author:** [@stillfreem](https://discuss.elastic.co/u/stillfreem)\ **Replies:** 2\ **Last updated:** [December 10, 2021, 11:49am UTC](https://discuss.elastic.co/t/prune-filter-whitelist-json-subfields-per-pipeline-log-file/291298 "2021-12-10T11:49:06Z")
Hi All, I've got a couple of questions! Can I whitelist only certain field from JSON using Prune? I saw similar topics suggesting ruby but wanted to avoid this as its a bit too complicated for me. For instance I'd li… --- ## [Logstash not creating new index](https://discuss.elastic.co/t/logstash-not-creating-new-index/291265)
**Author:** [@will.nickisch](https://discuss.elastic.co/u/will.nickisch)\ **Replies:** 7\ **Last updated:** [December 9, 2021, 10:28pm UTC](https://discuss.elastic.co/t/logstash-not-creating-new-index/291265 "2021-12-09T22:28:02Z")
I am having an issue with multiple tcp inputs and logstash not pulling in messages from the second tcp input. The data received on port 6050 is going into the default index logstash-%{+YYYY.MM.dd} but the data sent from… --- ## [Logstash arbitrarily hangs with SIGINT. Happens roughly every few days](https://discuss.elastic.co/t/logstash-arbitrarily-hangs-with-sigint-happens-roughly-every-few-days/291258)
**Author:** [@jtocher](https://discuss.elastic.co/u/jtocher)\ **Replies:** 1\ **Last updated:** [December 9, 2021, 8:51pm UTC](https://discuss.elastic.co/t/logstash-arbitrarily-hangs-with-sigint-happens-roughly-every-few-days/291258 "2021-12-09T20:51:43Z")
We have a Logstash server as part of our Elastic Stack on Windows. About once a week, logs stop ingesting and the culprit is always Logstash. Restarting the service on that VM fixes the issue, but I'd like to find the pr… --- ## [Logstash Can't read the elasticsearch username and password from Keystore!](https://discuss.elastic.co/t/logstash-cant-read-the-elasticsearch-username-and-password-from-keystore/291350)
**Author:** [@Maryam2021](https://discuss.elastic.co/u/Maryam2021)\ **Replies:** 0\ **Last updated:** [December 9, 2021, 4:39pm UTC](https://discuss.elastic.co/t/logstash-cant-read-the-elasticsearch-username-and-password-from-keystore/291350 "2021-12-09T16:39:32Z")
Hello All, I'm trying to build ELK stack on EC2 instances. I have built the ES cluster, Kibana and Logstash. The cluster and Kibana are happy and talking to each other fine but can't figure out the keystore issue with L… --- ## [Metric data from csv](https://discuss.elastic.co/t/metric-data-from-csv/291314)
**Author:** [@sirReeall](https://discuss.elastic.co/u/sirReeall)\ **Replies:** 1\ **Last updated:** [December 9, 2021, 2:52pm UTC](https://discuss.elastic.co/t/metric-data-from-csv/291314 "2021-12-09T14:52:11Z")
Hello, I'm trying to import metrics data from a CSV file. Currently I'm using a csv filter like: filter { csv { autodetect\_column\_names =\> true } date { match =\> \["\[t\]","UNIX"\] } mutat… --- ## [Parse json in Log field to get individual fields for visualization](https://discuss.elastic.co/t/parse-json-in-log-field-to-get-individual-fields-for-visualization/291244)
**Author:** [@girija](https://discuss.elastic.co/u/girija)\ **Replies:** 7\ **Last updated:** [December 9, 2021, 12:24pm UTC](https://discuss.elastic.co/t/parse-json-in-log-field-to-get-individual-fields-for-visualization/291244 "2021-12-09T12:24:05Z")
Hello I am new to Kibana and have difficulty reading the logs for visualization. I have pasted a sample of how my log field looks. I need to read the values inside the log for creating visualizations in Kibana, like w… --- ## [Logstash issues with certificate](https://discuss.elastic.co/t/logstash-issues-with-certificate/291124)
**Author:** [@dominbdg](https://discuss.elastic.co/u/dominbdg)\ **Replies:** 4\ **Last updated:** [December 9, 2021, 12:11pm UTC](https://discuss.elastic.co/t/logstash-issues-with-certificate/291124 "2021-12-09T12:11:10Z")
Hello, From a couple of days I'm fighting with certificates for logstash. Certificate was generated for logstash and filebeat clients but after couple of minutes I'm receiving below errors: in attach I don't know if t… --- ## [ELK stack setup](https://discuss.elastic.co/t/elk-stack-setup/291292)
**Author:** [@girija](https://discuss.elastic.co/u/girija)\ **Replies:** 2\ **Last updated:** [December 9, 2021, 9:20am UTC](https://discuss.elastic.co/t/elk-stack-setup/291292 "2021-12-09T09:20:29Z")
Hello I am new to the ELK world and find myself confused. Although my stack is working fine and I am able to get the logs I want, I am not sure about what all is really required for correct setup and if I have anything … --- ## [Is this a right filter?](https://discuss.elastic.co/t/is-this-a-right-filter/291118)
**Author:** [@smam](https://discuss.elastic.co/u/smam)\ **Replies:** 4\ **Last updated:** [December 9, 2021, 9:18am UTC](https://discuss.elastic.co/t/is-this-a-right-filter/291118 "2021-12-09T09:18:03Z")
filter { if "warning" in "message" { mutate { add\_field =\> { "error-field" =\> "An error occured" } } } } Is this right? Because it seems to not be working. If I open Elasticsearch and… --- ## [Is posible to set a text as optional in RegularExpresions?](https://discuss.elastic.co/t/is-posible-to-set-a-text-as-optional-in-regularexpresions/291114)
**Author:** [@rodri.gz](https://discuss.elastic.co/u/rodri.gz)\ **Replies:** 2\ **Last updated:** [December 9, 2021, 8:22am UTC](https://discuss.elastic.co/t/is-posible-to-set-a-text-as-optional-in-regularexpresions/291114 "2021-12-09T08:22:43Z")
Hola! Se puede poner textos como opcionales en regex? these days im reciving some logs with this format: \[07/12/2021 09:59:33 \[Id XXXXXXXXXXXXXXXX\] Parameters send to YYYYYY: action1: AAAAAAAAAAAA action2: BBBBBBBBB… --- ## [Is posible to have iptional field with regex as grok optionals fields?](https://discuss.elastic.co/t/is-posible-to-have-iptional-field-with-regex-as-grok-optionals-fields/290645)
**Author:** [@rodri.gz](https://discuss.elastic.co/u/rodri.gz)\ **Replies:** 5\ **Last updated:** [December 9, 2021, 8:18am UTC](https://discuss.elastic.co/t/is-posible-to-have-iptional-field-with-regex-as-grok-optionals-fields/290645 "2021-12-09T08:18:25Z")
Hello! i want to know if a could set a field as optional in regular expresions as i do with grok for example: (%{DATA:Message})? Thank you! --- ## [Unable to create index on elasticsearch](https://discuss.elastic.co/t/unable-to-create-index-on-elasticsearch/290657)
**Author:** [@Somesh\_ng](https://discuss.elastic.co/u/Somesh_ng)\ **Replies:** 5\ **Last updated:** [December 9, 2021, 7:27am UTC](https://discuss.elastic.co/t/unable-to-create-index-on-elasticsearch/290657 "2021-12-09T07:27:47Z")
I'm trying to ingest the data into Elasticsearch using logstash http\_poller but the index is not getting created in Elasticsearch. Can someone help me on this input { http\_poller { urls =\> { "te… --- ## [No detailed errors in logstash](https://discuss.elastic.co/t/no-detailed-errors-in-logstash/291233)
**Author:** [@SSivaji](https://discuss.elastic.co/u/SSivaji)\ **Replies:** 2\ **Last updated:** [December 9, 2021, 3:05am UTC](https://discuss.elastic.co/t/no-detailed-errors-in-logstash/291233 "2021-12-09T03:05:58Z")
When I try to Index a specific text. Able to index a in devtools, but when the same text is provided via logstash it is not getting Indexed, enabled debug logs it just prints the text to console, but not getting Indexe… --- ## [Processing Multiple Rows From JDBC Streaming](https://discuss.elastic.co/t/processing-multiple-rows-from-jdbc-streaming/291012)
**Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 22\ **Last updated:** [December 8, 2021, 8:37pm UTC](https://discuss.elastic.co/t/processing-multiple-rows-from-jdbc-streaming/291012 "2021-12-08T20:37:51Z")
I'm querying a database to get questions asked in a ticket. The query asks for all questions asked by the ticket id. So a single query may return multiple rows. I'd like to be able to parse them all into separate fiel… --- ## [CSV Illegal Quoting in Line 1](https://discuss.elastic.co/t/csv-illegal-quoting-in-line-1/289834)
**Author:** [@Micah\_Barsness](https://discuss.elastic.co/u/Micah_Barsness)\ **Replies:** 13\ **Last updated:** [December 8, 2021, 8:19pm UTC](https://discuss.elastic.co/t/csv-illegal-quoting-in-line-1/289834 "2021-12-08T20:19:33Z")
I'm receiving the following error & some of my incoming logs are being not being parsed and therefore dropped. \[2021-11-22T09:50:11,381\]\[WARN \]\[logstash.filters.csv \]\[MID-CUCM\]\[6a45eeb30e5ba587e03ab6fc5b123b705e6e76… --- ## [Imap input gives weird error](https://discuss.elastic.co/t/imap-input-gives-weird-error/290671)
**Author:** [@zozo6015](https://discuss.elastic.co/u/zozo6015)\ **Replies:** 6\ **Last updated:** [December 8, 2021, 5:16pm UTC](https://discuss.elastic.co/t/imap-input-gives-weird-error/290671 "2021-12-08T17:16:38Z")
Hello, I want to run imap plugin to load some reporting mails into Elasticsearch. I have it configured as the following: input { imap { check\_interval =\> "10" host =\> "host" port =\> "995" user =\> "em… --- ## [Parsing issues](https://discuss.elastic.co/t/parsing-issues/291159)
**Author:** [@nickel43](https://discuss.elastic.co/u/nickel43)\ **Replies:** 6\ **Last updated:** [December 8, 2021, 4:32pm UTC](https://discuss.elastic.co/t/parsing-issues/291159 "2021-12-08T16:32:13Z")
Hello, I have an issue with parsing my data. Here is a sample of JSON: { "info": { "generated\_on": "2017-12-03 08:41:42.057563", "slice": "0-999", "version": "v1" }, "playlists":… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=174) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=176)