# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=123 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 124 --- ## [Removing square brackets from json key in http input](https://discuss.elastic.co/t/removing-square-brackets-from-json-key-in-http-input/311243)
**Author:** [@Felipe\_Fuller](https://discuss.elastic.co/u/Felipe_Fuller)\ **Replies:** 5\ **Last updated:** [August 3, 2022, 2:51pm UTC](https://discuss.elastic.co/t/removing-square-brackets-from-json-key-in-http-input/311243 "2022-08-03T14:51:09Z")
Hi Community! I've been having the following problem. I'm receiving, through http input filter, a JSON that has bracket square in the keys so Logstash isn't able to parse it, as result it returns a \_jsonparsefailure. A… --- ## [Logstash overfilling Persistent Queue?](https://discuss.elastic.co/t/logstash-overfilling-persistent-queue/310197)
**Author:** [@timcorwin](https://discuss.elastic.co/u/timcorwin)\ **Replies:** 3\ **Last updated:** [August 3, 2022, 2:44pm UTC](https://discuss.elastic.co/t/logstash-overfilling-persistent-queue/310197 "2022-08-03T14:44:03Z")
I am running the logstash:7.17.4 docker image, and I've set queue.type: persisted in my logstash.yml, leaving all other queue settings at the default. Most of the time, I see the normal, expected behavior, where my even… --- ## [Cut off matching content and write to field](https://discuss.elastic.co/t/cut-off-matching-content-and-write-to-field/311197)
**Author:** [@anon90868141](https://discuss.elastic.co/u/anon90868141)\ **Replies:** 3\ **Last updated:** [August 3, 2022, 11:44am UTC](https://discuss.elastic.co/t/cut-off-matching-content-and-write-to-field/311197 "2022-08-03T11:44:02Z")
Hi, is there any way to match a pattern inside a log and just write a port of it to a field? Here's a abstraction of my log: elapsedTimeMs=41 I'd wrap it in () because it's sporadic and want the 41 to be written into … --- ## [Logstash 8.3.3 removes entries from sincedb](https://discuss.elastic.co/t/logstash-8-3-3-removes-entries-from-sincedb/311293)
**Author:** [@ylasri](https://discuss.elastic.co/u/ylasri)\ **Replies:** 0\ **Last updated:** [August 3, 2022, 10:37am UTC](https://discuss.elastic.co/t/logstash-8-3-3-removes-entries-from-sincedb/311293 "2022-08-03T10:37:24Z")
Hello, I have a pipeline using Logstash 8.3.3 The pipeline is consuming csv file from an input folder using file input The csv files are produced by an other system on the flight This my input input { file { p… --- ## [What is the best practice for multiple table? Multiple logstash config or single config?](https://discuss.elastic.co/t/what-is-the-best-practice-for-multiple-table-multiple-logstash-config-or-single-config/311277)
**Author:** [@myx](https://discuss.elastic.co/u/myx)\ **Replies:** 0\ **Last updated:** [August 3, 2022, 8:38am UTC](https://discuss.elastic.co/t/what-is-the-best-practice-for-multiple-table-multiple-logstash-config-or-single-config/311277 "2022-08-03T08:38:52Z")
Hi, I just learned ELK and I'm confused about this use case for its Logstash implementation for a project that has quite a bad DB design... So I have index in Elasticsearch called book. Inside it, there's a short infor… --- ## [Jdbc\_driver\_library, file not readable (please check user and group permissions for the path)](https://discuss.elastic.co/t/jdbc-driver-library-file-not-readable-please-check-user-and-group-permissions-for-the-path/311260)
**Author:** [@Kannanmk](https://discuss.elastic.co/u/Kannanmk)\ **Replies:** 0\ **Last updated:** [August 3, 2022, 5:25am UTC](https://discuss.elastic.co/t/jdbc-driver-library-file-not-readable-please-check-user-and-group-permissions-for-the-path/311260 "2022-08-03T05:25:33Z")
hi @guyboertje here i am trying to update my config into AWS but it shows "Error: unable to load /home/ubuntu/Jebin/Corban\_Project\_Docker\_21\_06\_2022/Corban\_Elastic\_Logstash\_Kibana/ELK\_Files/postgresql-42.4.0.jar from… --- ## [Possible race condition causing Elasticsearch output plugin to overwrite supplied index with rollover alias](https://discuss.elastic.co/t/possible-race-condition-causing-elasticsearch-output-plugin-to-overwrite-supplied-index-with-rollover-alias/311174)
**Author:** [@andreastoom](https://discuss.elastic.co/u/andreastoom)\ **Replies:** 2\ **Last updated:** [August 3, 2022, 5:18am UTC](https://discuss.elastic.co/t/possible-race-condition-causing-elasticsearch-output-plugin-to-overwrite-supplied-index-with-rollover-alias/311174 "2022-08-03T05:18:41Z")
Hi, Since migrating to using data streams we have started to see that a subset of our data all of a sudden are being written to an index called "logstash" instead of the data stream. After a bit of digging around it see… --- ## [Filebeat not working for k8s ingress nginx controller](https://discuss.elastic.co/t/filebeat-not-working-for-k8s-ingress-nginx-controller/311259)
**Author:** [@SABER\_Ye](https://discuss.elastic.co/u/SABER_Ye)\ **Replies:** 0\ **Last updated:** [August 3, 2022, 5:17am UTC](https://discuss.elastic.co/t/filebeat-not-working-for-k8s-ingress-nginx-controller/311259 "2022-08-03T05:17:06Z")
filebeat config: --- apiVersion: v1 kind: ConfigMap metadata: name: filebeat-config namespace: kube-system labels: k8s-app: filebeat data: filebeat.yml: |- filebeat.inputs: - type: log en… --- ## [Error while parsing nested json on filebeat](https://discuss.elastic.co/t/error-while-parsing-nested-json-on-filebeat/311256)
**Author:** [@RaonyO](https://discuss.elastic.co/u/RaonyO)\ **Replies:** 0\ **Last updated:** [August 3, 2022, 4:15am UTC](https://discuss.elastic.co/t/error-while-parsing-nested-json-on-filebeat/311256 "2022-08-03T04:15:59Z")
I have the following error on logstash when trying to index json nested logs that are inside square brackets '' Error parsing json {:source=\>"message", :raw=\>"\]", :exception=\>#\ **Author:** [@grvGarg](https://discuss.elastic.co/u/grvGarg)\ **Replies:** 4\ **Last updated:** [August 3, 2022, 3:52am UTC](https://discuss.elastic.co/t/not-able-to-use-postgres-query-with-double-quotes-in-jdbc-streaming-plugin/311141 "2022-08-03T03:52:00Z") Hi I am using jdbc\_streaming plugin. I have a complex query and also my postgres required double quotes (") with each table/column name. I am also using Case When in my query, which requires apostrophe('). Plugin does… --- ## [JDBC Plugin — Missing Converter handling for full class name=org.postgresql.util.PGobject, simple name=PGobject](https://discuss.elastic.co/t/jdbc-plugin-missing-converter-handling-for-full-class-name-org-postgresql-util-pgobject-simple-name-pgobject/311246)
**Author:** [@5k\_pwc](https://discuss.elastic.co/u/5k_pwc)\ **Replies:** 0\ **Last updated:** [August 2, 2022, 10:30pm UTC](https://discuss.elastic.co/t/jdbc-plugin-missing-converter-handling-for-full-class-name-org-postgresql-util-pgobject-simple-name-pgobject/311246 "2022-08-02T22:30:21Z")
hi, my table geometry, JSONB, i get the below exception. Exception when executing JDBC query {:exception=\>Sequel::DatabaseError, :message=\>"Java::OrgLogstash::MissingConverterException: Missing Converter handling for f… --- ## [Logstash Output doesnt have grok fields but Event.original only](https://discuss.elastic.co/t/logstash-output-doesnt-have-grok-fields-but-event-original-only/311220)
**Author:** [@vsudula](https://discuss.elastic.co/u/vsudula)\ **Replies:** 7\ **Last updated:** [August 2, 2022, 6:30pm UTC](https://discuss.elastic.co/t/logstash-output-doesnt-have-grok-fields-but-event-original-only/311220 "2022-08-02T18:30:18Z")
I am using Logstash 8.3 and i have provided my logs below . When logstsh ingests the data to elastic its ingesting data as whole line as text instead of individual values that i defined in grok pattern. //My Code here. … --- ## [Missing RSS Items](https://discuss.elastic.co/t/missing-rss-items/311199)
**Author:** [@pdziumla](https://discuss.elastic.co/u/pdziumla)\ **Replies:** 1\ **Last updated:** [August 2, 2022, 4:48pm UTC](https://discuss.elastic.co/t/missing-rss-items/311199 "2022-08-02T16:48:17Z")
Hello Guys, I'm completely new to RSS, XML and that stuff so I might overlook something quite simple here. So, here ist my problem: I succesfully setup the RSS-Plugin and it's working fine. Except it seem to miss out … --- ## [Extract time from txt file and date from file name and add them to @timestamp field](https://discuss.elastic.co/t/extract-time-from-txt-file-and-date-from-file-name-and-add-them-to-timestamp-field/311196)
**Author:** [@Faiz\_Shamri](https://discuss.elastic.co/u/Faiz_Shamri)\ **Replies:** 1\ **Last updated:** [August 2, 2022, 4:35pm UTC](https://discuss.elastic.co/t/extract-time-from-txt-file-and-date-from-file-name-and-add-them-to-timestamp-field/311196 "2022-08-02T16:35:30Z")
I am trying to extract time field from txt file which looks like: 14:39:38 someName clientName 968112300 1008008000 39895700 also tyring to extract date from file name /path/to/file/name/filename\_2… --- ## [Overwrite @timestamp filed by date and time filed in the CSV file](https://discuss.elastic.co/t/overwrite-timestamp-filed-by-date-and-time-filed-in-the-csv-file/311142)
**Author:** [@Faiz\_Shamri](https://discuss.elastic.co/u/Faiz_Shamri)\ **Replies:** 7\ **Last updated:** [August 2, 2022, 2:42pm UTC](https://discuss.elastic.co/t/overwrite-timestamp-filed-by-date-and-time-filed-in-the-csv-file/311142 "2022-08-02T14:42:43Z")
My logstash conf. file is like the below: input { file { path =\> "/somepath/balance\_20220731.csv" start\_position =\> "beginning" sincedb\_path =\> "/dev/null" } } filter { csv { separator =\> ";" … --- ## [Logs with the same timestamp wrong located](https://discuss.elastic.co/t/logs-with-the-same-timestamp-wrong-located/311207)
**Author:** [@TaiNguyen](https://discuss.elastic.co/u/TaiNguyen)\ **Replies:** 0\ **Last updated:** [August 2, 2022, 11:25am UTC](https://discuss.elastic.co/t/logs-with-the-same-timestamp-wrong-located/311207 "2022-08-02T11:25:28Z")
Hello! Im using logstash version 7.10.3. Below is my logstash filter config for timstamp match filter { if (\[biz\] and \[log\]\[type\] != "nginx") { grok { match =\> { "messag… --- ## [Logstash s3 output](https://discuss.elastic.co/t/logstash-s3-output/311118)
**Author:** [@Giridharan\_C](https://discuss.elastic.co/u/Giridharan_C)\ **Replies:** 4\ **Last updated:** [August 2, 2022, 8:59am UTC](https://discuss.elastic.co/t/logstash-s3-output/311118 "2022-08-02T08:59:12Z")
My logstash s3 output is not working as defined in the pipeline config. The exact output config is working for other pipelines. The dynamic directory structure(/syslog/%{+YYYY}/%{+MM}/%{+dd}/%{+HH}) defined in the config… --- ## [Convert dates in EJSON format / Access current processed field alias](https://discuss.elastic.co/t/convert-dates-in-ejson-format-access-current-processed-field-alias/311114)
**Author:** [@thomas\_seres\_panda](https://discuss.elastic.co/u/thomas_seres_panda)\ **Replies:** 0\ **Last updated:** [August 1, 2022, 12:26pm UTC](https://discuss.elastic.co/t/convert-dates-in-ejson-format-access-current-processed-field-alias/311114 "2022-08-01T12:26:39Z")
Hi ! I would like to dynamically : match any fields corresponding to a format like "yyyy-MM-dd" add new field by using current processed field : example : add\_field =\> { "\[%{current\_processed\_field\_name}'\_ejson'\]\[$… --- ## [Logstash if condition regex not working](https://discuss.elastic.co/t/logstash-if-condition-regex-not-working/310737)
**Author:** [@Gosia96](https://discuss.elastic.co/u/Gosia96)\ **Replies:** 10\ **Last updated:** [August 1, 2022, 1:56pm UTC](https://discuss.elastic.co/t/logstash-if-condition-regex-not-working/310737 "2022-08-01T13:56:53Z")
Hi, I'm trying to use regex in the if condition in logstash. My goal is to send the output into 2 different indexes basing on what the field "tags" contains. So far I had no luck with it. Logstash simply ignores my co… --- ## [Grok Plugin: load custom patterns in Oniguruma format from file](https://discuss.elastic.co/t/grok-plugin-load-custom-patterns-in-oniguruma-format-from-file/311108)
**Author:** [@who](https://discuss.elastic.co/u/who)\ **Replies:** 0\ **Last updated:** [August 1, 2022, 11:52am UTC](https://discuss.elastic.co/t/grok-plugin-load-custom-patterns-in-oniguruma-format-from-file/311108 "2022-08-01T11:52:44Z")
According to the documentation, we can use Oniguruma regex pattern formats in grok plugin (i.e. (?\the pattern here)). Also, there's the patterns\_dir configuration option that instructs grok filter to load pa… --- ## [Display all python prints in logstash](https://discuss.elastic.co/t/display-all-python-prints-in-logstash/310351)
**Author:** [@Didi\_Sisi](https://discuss.elastic.co/u/Didi_Sisi)\ **Replies:** 1\ **Last updated:** [August 1, 2022, 11:39am UTC](https://discuss.elastic.co/t/display-all-python-prints-in-logstash/310351 "2022-08-01T11:39:30Z")
Hello, I have a python script like this: import time dictionary1 ={ "id": "02", "name": "sunil", "department": "HR" } dictionary2 ={ "id": "03", "name": "sunilals", "depa… --- ## [Logstash Translate filter plugin based on multilevel dictionary](https://discuss.elastic.co/t/logstash-translate-filter-plugin-based-on-multilevel-dictionary/311105)
**Author:** [@who](https://discuss.elastic.co/u/who)\ **Replies:** 0\ **Last updated:** [August 1, 2022, 10:50am UTC](https://discuss.elastic.co/t/logstash-translate-filter-plugin-based-on-multilevel-dictionary/311105 "2022-08-01T10:50:01Z")
Assuming we have a dictionary with nested structure (YAML): "key1": "sub-key1": "value1" "sub-key2": "value2" "sub-key3": "value3" "key2": "sub-key4": "value4" "sub-key5": "value5" "sub-key6": "value6" Curr… --- ## [Reference to grok patterns that are in another field of event](https://discuss.elastic.co/t/reference-to-grok-patterns-that-are-in-another-field-of-event/311104)
**Author:** [@who](https://discuss.elastic.co/u/who)\ **Replies:** 0\ **Last updated:** [August 1, 2022, 10:46am UTC](https://discuss.elastic.co/t/reference-to-grok-patterns-that-are-in-another-field-of-event/311104 "2022-08-01T10:46:15Z")
There's a regexes array field in the event that contains some regexes: "regexes" =\> \[ \[0\] "regex1", \[1\] "regex2", \[2\] "regex3" \] As this field's content get filled dynamicall… --- ## [Logstash Ruby concat & event](https://discuss.elastic.co/t/logstash-ruby-concat-event/308816)
**Author:** [@jsauvebois](https://discuss.elastic.co/u/jsauvebois)\ **Replies:** 3\ **Last updated:** [August 1, 2022, 9:42am UTC](https://discuss.elastic.co/t/logstash-ruby-concat-event/308816 "2022-08-01T09:42:24Z")
Hi, In a first time, I tested how to add a field via ruby code based on existing field duration and event ... ruby { code =\> " if event.get('duration').nil? event.set('duration\_time','') else duration\_time = ev… --- ## [Logstash: dynamic topic based on value of field](https://discuss.elastic.co/t/logstash-dynamic-topic-based-on-value-of-field/310976)
**Author:** [@jori-be](https://discuss.elastic.co/u/jori-be)\ **Replies:** 2\ **Last updated:** [August 1, 2022, 9:10am UTC](https://discuss.elastic.co/t/logstash-dynamic-topic-based-on-value-of-field/310976 "2022-08-01T09:10:27Z")
I'm trying to configure Logstash with Kafka output. Basically I want to put data in different Kafka topics based on a field. So in example below, I have a field name in the data. All documents should be posted to the to… --- ## [Logstash 8 @timestamp field format was changed to microseconds percision](https://discuss.elastic.co/t/logstash-8-timestamp-field-format-was-changed-to-microseconds-percision/310852)
**Author:** [@Ofir\_Edi](https://discuss.elastic.co/u/Ofir_Edi)\ **Replies:** 7\ **Last updated:** [August 1, 2022, 7:23am UTC](https://discuss.elastic.co/t/logstash-8-timestamp-field-format-was-changed-to-microseconds-percision/310852 "2022-08-01T07:23:38Z")
Hi, I have been using Logstash for a while now and when upgrading to version 8 I can see the @timestamp field format was changed from milliseconds percision to microseconds percision (meaning instead of 2022-07-28T09:46… --- ## [Logstash stuck after Restart/Stop command](https://discuss.elastic.co/t/logstash-stuck-after-restart-stop-command/310797)
**Author:** [@Oriya](https://discuss.elastic.co/u/Oriya)\ **Replies:** 2\ **Last updated:** [July 31, 2022, 9:48pm UTC](https://discuss.elastic.co/t/logstash-stuck-after-restart-stop-command/310797 "2022-07-31T21:48:30Z")
Hi, i have a logstash server (8.1.3 version) with 3 pipelines. each pipeline configure with jdbc to collect data from sql server. everything works ok until i try to execute the : "systemctl restart logstash" and it'… --- ## [Errno::EACCES: Permission denied - NUL not fixed with enableADS](https://discuss.elastic.co/t/errno-permission-denied-nul-not-fixed-with-enableads/310846)
**Author:** [@Bert\_Van\_der\_Heyden](https://discuss.elastic.co/u/Bert_Van_der_Heyden)\ **Replies:** 3\ **Last updated:** [July 31, 2022, 9:03pm UTC](https://discuss.elastic.co/t/errno-permission-denied-nul-not-fixed-with-enableads/310846 "2022-07-31T21:03:35Z")
Running logstash (latest version 8.3.2) on Windows gives me: Errno::EACCES: Permission denied - NUL sysopen at org/jruby/RubyIO.java:1237 Found out that this is a known issue and fixed with -Djdk.io.File.enableADS=tru… --- ## [Logstash not starting; error reason : Your settings are invalid](https://discuss.elastic.co/t/logstash-not-starting-error-reason-your-settings-are-invalid/310440)
**Author:** [@sanuboy](https://discuss.elastic.co/u/sanuboy)\ **Replies:** 7\ **Last updated:** [July 30, 2022, 5:36pm UTC](https://discuss.elastic.co/t/logstash-not-starting-error-reason-your-settings-are-invalid/310440 "2022-07-30T17:36:38Z")
Hi, This is the first time i am trying to setup ELK stack to read my application log files and i have been struggling to get through this issue for about two days and finally resorted to asking for help. I have illustrat… --- ## [Trying to make logstash work with multiline logs on a SIEM platform](https://discuss.elastic.co/t/trying-to-make-logstash-work-with-multiline-logs-on-a-siem-platform/310951)
**Author:** [@currybread](https://discuss.elastic.co/u/currybread)\ **Replies:** 1\ **Last updated:** [July 30, 2022, 12:49pm UTC](https://discuss.elastic.co/t/trying-to-make-logstash-work-with-multiline-logs-on-a-siem-platform/310951 "2022-07-30T12:49:23Z")
Situation: Multiline logs collected are sent to a SIEM event collector via Logstash but having issues test.log: \[5/8/22 7:31:23:546 SGT\] FFDC Exception:java.io.FileNotFoundException SourceId:com.ibm.ws.webcontaine… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=122) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=124)