# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=199 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 200 --- ## [Index not creating](https://discuss.elastic.co/t/index-not-creating/282345)
**Author:** [@jubin03](https://discuss.elastic.co/u/jubin03)\ **Replies:** 17\ **Last updated:** [August 26, 2021, 10:55am UTC](https://discuss.elastic.co/t/index-not-creating/282345 "2021-08-26T10:55:13Z")
After configuring filter.conf and logstash input index not showing in Kibana. Successfully tested logstash conf files. No errors. Below are the conf files Filter.conf filter { if \[type\] == "signinattempts" { j… --- ## [Blacklisting with Ruby Filter](https://discuss.elastic.co/t/blacklisting-with-ruby-filter/282517)
**Author:** [@pkward](https://discuss.elastic.co/u/pkward)\ **Replies:** 2\ **Last updated:** [August 26, 2021, 12:28am UTC](https://discuss.elastic.co/t/blacklisting-with-ruby-filter/282517 "2021-08-26T00:28:47Z")
Hi, I'm looking to blacklist 1000+ nested fields. I've read in another forum that it's not possible to use the prune filter for nested filters, but I can with a ruby filter. I wanted to know is there a way to blacklist … --- ## [Fingerprint & record drop not working as expected](https://discuss.elastic.co/t/fingerprint-record-drop-not-working-as-expected/282513)
**Author:** [@GitSpree23](https://discuss.elastic.co/u/GitSpree23)\ **Replies:** 4\ **Last updated:** [August 25, 2021, 6:35pm UTC](https://discuss.elastic.co/t/fingerprint-record-drop-not-working-as-expected/282513 "2021-08-25T18:35:05Z")
I'm reading lines from a log that, after converting to JSON need to be split based on a field & then I want to generate fingerprint using some fields & drop the event if the fields are not found. But I'm facing issues: … --- ## [Nanoseconds with Logstash and Elasticsearch](https://discuss.elastic.co/t/nanoseconds-with-logstash-and-elasticsearch/282492)
**Author:** [@Nico\_Pampaloni](https://discuss.elastic.co/u/Nico_Pampaloni)\ **Replies:** 7\ **Last updated:** [August 25, 2021, 4:21pm UTC](https://discuss.elastic.co/t/nanoseconds-with-logstash-and-elasticsearch/282492 "2021-08-25T16:21:43Z")
Good Morning! I am a little stuck figuring out how to handle custom time formats with nanosecond precision. The data that I am currently trying to ingest has two different time formats: 1629480840.652062565 and 2021:08:2… --- ## [Aggregate plugin does not add all results](https://discuss.elastic.co/t/aggregate-plugin-does-not-add-all-results/282503)
**Author:** [@josedelrio](https://discuss.elastic.co/u/josedelrio)\ **Replies:** 2\ **Last updated:** [August 25, 2021, 4:20pm UTC](https://discuss.elastic.co/t/aggregate-plugin-does-not-add-all-results/282503 "2021-08-25T16:20:16Z")
Aggreation plugin overwrites data I am trying to implement an aggregation using a MySQL 5.7 database as source. select u.id as user\_id, u.first\_name as first\_name, u.last\_name as last\_name,t.id as 'tag\_id', t.name as 't… --- ## [Logstash jdbc\_static types doesn't work (longtext, tinyint...)](https://discuss.elastic.co/t/logstash-jdbc-static-types-doesnt-work-longtext-tinyint/282497)
**Author:** [@Atesrin](https://discuss.elastic.co/u/Atesrin)\ **Replies:** 2\ **Last updated:** [August 25, 2021, 2:58pm UTC](https://discuss.elastic.co/t/logstash-jdbc-static-types-doesnt-work-longtext-tinyint/282497 "2021-08-25T14:58:33Z")
Hi, I would like to know if there is some place where I can find some information about the accepted types for jdbc\_static ? From what I see on websites only "varchar" seems accepted, but I have some longtext, tinyint … --- ## [Token input in Logstash](https://discuss.elastic.co/t/token-input-in-logstash/282460)
**Author:** [@sama.sowjanya](https://discuss.elastic.co/u/sama.sowjanya)\ **Replies:** 1\ **Last updated:** [August 25, 2021, 2:22pm UTC](https://discuss.elastic.co/t/token-input-in-logstash/282460 "2021-08-25T14:22:05Z")
We have an API from Genesys to integrate with Elasticsearch through logstash, but the authorization to that api is token system. Is there a way to fetch the token in Logstash. What would be the best approach to integrat… --- ## [Understanding ILM policy](https://discuss.elastic.co/t/understanding-ilm-policy/282303)
**Author:** [@selin](https://discuss.elastic.co/u/selin)\ **Replies:** 4\ **Last updated:** [August 25, 2021, 3:24am UTC](https://discuss.elastic.co/t/understanding-ilm-policy/282303 "2021-08-25T03:24:40Z")
Hi friends,, It is noted i can set ILM policy in logstash attach to index output { elasticsearch { ilm\_rollover\_alias =\> "custom" ilm\_pattern =\> "000001" ilm\_policy =\> "custom\_policy" … --- ## [Problem with if statement in GROK filter](https://discuss.elastic.co/t/problem-with-if-statement-in-grok-filter/282466)
**Author:** [@vp096](https://discuss.elastic.co/u/vp096)\ **Replies:** 4\ **Last updated:** [August 25, 2021, 11:27am UTC](https://discuss.elastic.co/t/problem-with-if-statement-in-grok-filter/282466 "2021-08-25T11:27:13Z")
Hello, I send syslog data from a WiFi controller to ElasticSearch. This controller sends me different types of packets and I want to keep only the packets of this type: Aug 25 11:19:50 2021 MC72-WI-AS1-06-B authmgr\[364… --- ## [Logstash Monitoring and AWS PrivateLink (Elastic Cloud)](https://discuss.elastic.co/t/logstash-monitoring-and-aws-privatelink-elastic-cloud/282338)
**Author:** [@nick\_cgi](https://discuss.elastic.co/u/nick_cgi)\ **Replies:** 3\ **Last updated:** [August 25, 2021, 10:16am UTC](https://discuss.elastic.co/t/logstash-monitoring-and-aws-privatelink-elastic-cloud/282338 "2021-08-25T10:16:24Z")
I have an Elastic Cloud instance, set to deploy in AWS (v7.13.x). Additionally, I have Logstash (v7.13.4) running within an AWS account, which is in the same region as the Elastic Cloud instance. In order to get Logsta… --- ## [Winlogbeat logs filtering in Logstash for storage management and relevant info to elastic](https://discuss.elastic.co/t/winlogbeat-logs-filtering-in-logstash-for-storage-management-and-relevant-info-to-elastic/282218)
**Author:** [@Rojal\_Paul](https://discuss.elastic.co/u/Rojal_Paul)\ **Replies:** 3\ **Last updated:** [August 25, 2021, 9:25am UTC](https://discuss.elastic.co/t/winlogbeat-logs-filtering-in-logstash-for-storage-management-and-relevant-info-to-elastic/282218 "2021-08-25T09:25:14Z")
I want to know the problem with my configuration. I want to reduce the windows logs saved in Logstash by sending relevant info such as warnings, errors. For now, it's taking a huge amount of data for storing all wind… --- ## [If else in logstash throwin error](https://discuss.elastic.co/t/if-else-in-logstash-throwin-error/282436)
**Author:** [@selin](https://discuss.elastic.co/u/selin)\ **Replies:** 2\ **Last updated:** [August 25, 2021, 9:06am UTC](https://discuss.elastic.co/t/if-else-in-logstash-throwin-error/282436 "2021-08-25T09:06:20Z")
Below is my logstash configuration file output { if "%{\[fields\]\[myvalue1\]}" == "value" and "%{\[fields\]\[myvalue2\]}" == "valu" { hosts =\> \["https://es1:9200","https://es2:9200","https://es3:9200"\] ilm\_enabled =\> … --- ## [Logstash not parse winlogbeat new version](https://discuss.elastic.co/t/logstash-not-parse-winlogbeat-new-version/282456)
**Author:** [@Nazakat](https://discuss.elastic.co/u/Nazakat)\ **Replies:** 0\ **Last updated:** [August 25, 2021, 7:49am UTC](https://discuss.elastic.co/t/logstash-not-parse-winlogbeat-new-version/282456 "2021-08-25T07:49:22Z")
There is create two fields and divided value version wies. Then i want to FileVersion field is remove --- ## [Logstash 7.9 JDBC- Oracle input - Session closing](https://discuss.elastic.co/t/logstash-7-9-jdbc-oracle-input-session-closing/282452)
**Author:** [@Prashant\_Achari](https://discuss.elastic.co/u/Prashant_Achari)\ **Replies:** 0\ **Last updated:** [August 25, 2021, 7:01am UTC](https://discuss.elastic.co/t/logstash-7-9-jdbc-oracle-input-session-closing/282452 "2021-08-25T07:01:58Z")
We have an scenario where we are connecting to oracle db for getting db stats using JDBC / Logstash. In some cases oracle db connection are not closed after query execution. These session are seen in long running sess… --- ## [Error using logstash : ParserError: Unexpected character ('\<'](https://discuss.elastic.co/t/error-using-logstash-parsererror-unexpected-character/281322)
**Author:** [@vp096](https://discuss.elastic.co/u/vp096)\ **Replies:** 18\ **Last updated:** [August 25, 2021, 5:54am UTC](https://discuss.elastic.co/t/error-using-logstash-parsererror-unexpected-character/281322 "2021-08-25T05:54:17Z")
Hello, I have been using ELK for a while and I am trying to retrieve the logs from a Wifi controller (Aruba Mobility Master). I checked with Wireshark, and I'm getting my syslog packets fine. However, logstash makes an … --- ## [ILM Index Lifecycle](https://discuss.elastic.co/t/ilm-index-lifecycle/282367)
**Author:** [@selin](https://discuss.elastic.co/u/selin)\ **Replies:** 2\ **Last updated:** [August 25, 2021, 4:23am UTC](https://discuss.elastic.co/t/ilm-index-lifecycle/282367 "2021-08-25T04:23:37Z")
Hi can i know if ilm\_rollover\_alias not speificed, will it take index name as alias? index =\> "my\_index" ilm\_pattern =\> "{now/d{yyyyMMdd}}-00001" so it will be my\_index-20210823-00001 ? --- ## [How to save the data obtained by using the http\_poller input plug-in of logstash to elasticsearch?](https://discuss.elastic.co/t/how-to-save-the-data-obtained-by-using-the-http-poller-input-plug-in-of-logstash-to-elasticsearch/282304)
**Author:** [@zhyp](https://discuss.elastic.co/u/zhyp)\ **Replies:** 5\ **Last updated:** [August 25, 2021, 3:06am UTC](https://discuss.elastic.co/t/how-to-save-the-data-obtained-by-using-the-http-poller-input-plug-in-of-logstash-to-elasticsearch/282304 "2021-08-25T03:06:01Z")
Hi :slightly\_smiling\_face: I use version 7.12.0. I used http\_poller to call the SQL API of elasticsearch and got some statistics. input { http\_poller { urls =\> { item =\> { method =\> post url =\> "http://localho… --- ## [S3 CSV Codec - Using a non printable character as separator](https://discuss.elastic.co/t/s3-csv-codec-using-a-non-printable-character-as-separator/282262)
**Author:** [@Hammond95](https://discuss.elastic.co/u/Hammond95)\ **Replies:** 4\ **Last updated:** [August 25, 2021, 12:55am UTC](https://discuss.elastic.co/t/s3-csv-codec-using-a-non-printable-character-as-separator/282262 "2021-08-25T00:55:47Z")
Is it possible to use a non printable character as a separator in the csv codec? codec =\> csv { columns =\> \["col1", "col2", "col3"\] charset =\> "UTF-8" separator =\> "\\\\u001F" } I have tried… --- ## [Best practices for logstash output configuration and Elasticsearch](https://discuss.elastic.co/t/best-practices-for-logstash-output-configuration-and-elasticsearch/281642)
**Author:** [@Vinay\_Kumar2](https://discuss.elastic.co/u/Vinay_Kumar2)\ **Replies:** 1\ **Last updated:** [August 24, 2021, 6:42pm UTC](https://discuss.elastic.co/t/best-practices-for-logstash-output-configuration-and-elasticsearch/281642 "2021-08-24T18:42:31Z")
Hello, My question is related to the correct way to configure the output, for Elasticsearch. Right now my Elasticsearch cluster is one dedicated master node, 3 data nodes and acts as master eligible nodes and 1 coordi… --- ## [Formatting data as a block in logstash](https://discuss.elastic.co/t/formatting-data-as-a-block-in-logstash/282027)
**Author:** [@selin](https://discuss.elastic.co/u/selin)\ **Replies:** 1\ **Last updated:** [August 24, 2021, 6:36pm UTC](https://discuss.elastic.co/t/formatting-data-as-a-block-in-logstash/282027 "2021-08-24T18:36:52Z")
Below is my sample data. How can it be filter based on start time and end time using grok pattern. Any idea friends since the data came as a stream of blocks. Can you guys suggest and provid example. I need to know start… --- ## [Grok filter](https://discuss.elastic.co/t/grok-filter/282374)
**Author:** [@lexanic](https://discuss.elastic.co/u/lexanic)\ **Replies:** 1\ **Last updated:** [August 24, 2021, 6:16pm UTC](https://discuss.elastic.co/t/grok-filter/282374 "2021-08-24T18:16:02Z")
Не работает grok filter grok { match =\> { "message" =\> \[ "%{POSTFIX\_MSG\_ID:taskid}: client=(?\\[^\]\]+)\[%{IP:client\_ip}\]", "%{POSTFIX\_MSG\_ID:taskid}: message-id=\<(?\\[^\>\]+)\>.", "%{POSTFIX\_MSG… --- ## [\[Logstash\] \ on json with \[null\]](https://discuss.elastic.co/t/logstash-java-31-nil-on-json-with-null/282382)
**Author:** [@zenkovac](https://discuss.elastic.co/u/zenkovac)\ **Replies:** 2\ **Last updated:** [August 24, 2021, 6:14pm UTC](https://discuss.elastic.co/t/logstash-java-31-nil-on-json-with-null/282382 "2021-08-24T18:14:51Z")
hi, i'm experiencing an error when processing a json log which has a \[null\] value on a field. ■The log is the following (tested as valid json): {"message":{"errorMessage":"INTERNAL\_SERVER\_ERROR","errorDetail":{"message… --- ## [S3 No files found in bucket](https://discuss.elastic.co/t/s3-no-files-found-in-bucket/282332)
**Author:** [@whassanwj](https://discuss.elastic.co/u/whassanwj)\ **Replies:** 4\ **Last updated:** [August 24, 2021, 3:33pm UTC](https://discuss.elastic.co/t/s3-no-files-found-in-bucket/282332 "2021-08-24T15:33:01Z")
Hi, I'm struggling with logstash conf to read from S3 bucket which has the latest date, example structure in S3 bucket as follows: apachelogs/web/2021-08-22/ apachelogs/web/2021-08-23/ apachelogs/web/2021-08-24/ Henc… --- ## [Trying to be ECS compliance in order to have visual analysis. It's hard! Help needed](https://discuss.elastic.co/t/trying-to-be-ecs-compliance-in-order-to-have-visual-analysis-its-hard-help-needed/282370)
**Author:** [@syunusic](https://discuss.elastic.co/u/syunusic)\ **Replies:** 0\ **Last updated:** [August 24, 2021, 3:04pm UTC](https://discuss.elastic.co/t/trying-to-be-ecs-compliance-in-order-to-have-visual-analysis-its-hard-help-needed/282370 "2021-08-24T15:04:13Z")
I’m renaming Fortigate’s field names from how Fortinet called them to a ECS compatible one. There is the integration section and there is also the exported field section. Which one do I have to follow? Three examples to … --- ## [Logstash KV filter - Wrong parsing](https://discuss.elastic.co/t/logstash-kv-filter-wrong-parsing/282333)
**Author:** [@vasek](https://discuss.elastic.co/u/vasek)\ **Replies:** 4\ **Last updated:** [August 24, 2021, 12:37pm UTC](https://discuss.elastic.co/t/logstash-kv-filter-wrong-parsing/282333 "2021-08-24T12:37:26Z")
Hello, I'm using KV filter in Logstash. Logstash configuration: filter { #PARSING LOG #------------------------------------------ grok { patterns\_dir =\> \["/etc/logstash/grok-patterns"\] break\_on\_mat… --- ## [Logstash http\_poller body escape double and single quotation marks](https://discuss.elastic.co/t/logstash-http-poller-body-escape-double-and-single-quotation-marks/282039)
**Author:** [@zhyp](https://discuss.elastic.co/u/zhyp)\ **Replies:** 2\ **Last updated:** [August 24, 2021, 1:38am UTC](https://discuss.elastic.co/t/logstash-http-poller-body-escape-double-and-single-quotation-marks/282039 "2021-08-24T01:38:21Z")
Hi I used logstash http\_ poller calls the SQL API of elasticsearch. There are double quotation marks and single quotation marks in the body. Here is the body I tested. body =\> '{"query": "SELECT \* FROM test WHERE name… --- ## [Data ingestion from MS SQL to Elastic search using logstash(on cloud deployement)](https://discuss.elastic.co/t/data-ingestion-from-ms-sql-to-elastic-search-using-logstash-on-cloud-deployement/282230)
**Author:** [@tushar6065](https://discuss.elastic.co/u/tushar6065)\ **Replies:** 1\ **Last updated:** [August 23, 2021, 10:02pm UTC](https://discuss.elastic.co/t/data-ingestion-from-ms-sql-to-elastic-search-using-logstash-on-cloud-deployement/282230 "2021-08-23T22:02:50Z")
I want to ingest data into Elastic search from MS SQL using Logstash and I was able to achieve this by setting up MS SQL JDBC plugin on my machine locally. But for upper environments, we will be going with Elastic Cloud … --- ## [Curl command to logstash](https://discuss.elastic.co/t/curl-command-to-logstash/282266)
**Author:** [@bamsom](https://discuss.elastic.co/u/bamsom)\ **Replies:** 1\ **Last updated:** [August 23, 2021, 9:35pm UTC](https://discuss.elastic.co/t/curl-command-to-logstash/282266 "2021-08-23T21:35:38Z")
I am running a curl command which gives me a numeric output. I have the input setup as exec \> command \> curl. I am outputting that data into Elastic search. I see the message as the numeric output. When I try to visualiz… --- ## [S3 Output Plugin: Correct Way to manage codec](https://discuss.elastic.co/t/s3-output-plugin-correct-way-to-manage-codec/282103)
**Author:** [@Hammond95](https://discuss.elastic.co/u/Hammond95)\ **Replies:** 1\ **Last updated:** [August 23, 2021, 5:21pm UTC](https://discuss.elastic.co/t/s3-output-plugin-correct-way-to-manage-codec/282103 "2021-08-23T17:21:01Z")
Hi everyone, I have the following problem: We have in place a pipeline which consist of: \[PrestoDB Clusters\] ==auditing==\> \[Kafka\] \<== \[Logstash\] ==\> Elastic + S3 The auditing messages on kafka are basically json mes… --- ## [Custom grok filter](https://discuss.elastic.co/t/custom-grok-filter/281105)
**Author:** [@gba8912](https://discuss.elastic.co/u/gba8912)\ **Replies:** 3\ **Last updated:** [August 23, 2021, 3:59pm UTC](https://discuss.elastic.co/t/custom-grok-filter/281105 "2021-08-23T15:59:51Z")
Hello, I'm new to all this so bear with me. My company has asked me to figure out elk stack and how to make it work for their logs. I have everything set up and working but I need to be able to parse the logs in kibana a… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=198) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=200)