# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=379 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 380 --- ## [CSV Date column replace to timestamp](https://discuss.elastic.co/t/csv-date-column-replace-to-timestamp/226560)
**Author:** [@lohit18](https://discuss.elastic.co/u/lohit18)\ **Replies:** 1\ **Last updated:** [April 5, 2020, 10:05am UTC](https://discuss.elastic.co/t/csv-date-column-replace-to-timestamp/226560 "2020-04-05T10:05:14Z")
Dear Team, Kindly help me out how to repalce with CSV date format(3/27/2020 18:01 PM) with logstash timestamp . I am passing .conf like this below input { file { path=\>"C:Downloads/low\_data.csv" start\_position=\>"… --- ## [I am unable to parse certain JSON messages](https://discuss.elastic.co/t/i-am-unable-to-parse-certain-json-messages/225888)
**Author:** [@Blason](https://discuss.elastic.co/u/Blason)\ **Replies:** 6\ **Last updated:** [April 5, 2020, 6:16am UTC](https://discuss.elastic.co/t/i-am-unable-to-parse-certain-json-messages/225888 "2020-04-05T06:16:48Z")
Hi Guys, I am trying to parse Json formatted messages certain messages are getting parsed properly; while certain are showing as jsonparsefailure. Then I checked the same message online json viewer and it gets parsed pe… --- ## [Unable to configure plugins](https://discuss.elastic.co/t/unable-to-configure-plugins/226526)
**Author:** [@jeffzhao](https://discuss.elastic.co/u/jeffzhao)\ **Replies:** 1\ **Last updated:** [April 4, 2020, 9:23pm UTC](https://discuss.elastic.co/t/unable-to-configure-plugins/226526 "2020-04-04T21:23:51Z")
Hi, I'm using Elasticsearch 1.7 and trying to copy documents from old index to new index using Logstash 7.6. Here is my configuration file: input { # We read from the "old" cluster elasticsearch { hosts =\> \[ "… --- ## [Sum with unique object](https://discuss.elastic.co/t/sum-with-unique-object/225425)
**Author:** [@Emna1](https://discuss.elastic.co/u/Emna1)\ **Replies:** 15\ **Last updated:** [April 4, 2020, 5:09pm UTC](https://discuss.elastic.co/t/sum-with-unique-object/225425 "2020-04-04T17:09:49Z")
For example if i have csv file with fields; Car, parking,date of entry V1, p1, 12-03-2020 12:30 V1, p1, 12-03-2020 11:30 V2, p2,12-03-2020 10:30 V2, p2,12-03-2020 10:45 How can i calculate number of time V1 enter… --- ## [Gsub remove square bracket](https://discuss.elastic.co/t/gsub-remove-square-bracket/226508)
**Author:** [@mattiacareddu](https://discuss.elastic.co/u/mattiacareddu)\ **Replies:** 2\ **Last updated:** [April 4, 2020, 4:23pm UTC](https://discuss.elastic.co/t/gsub-remove-square-bracket/226508 "2020-04-04T16:23:43Z")
I'm trying to remove square bracket from my string using gsub filter. The string is something like this: '\["/path/to/1","/path/to/2"\]' I tried different ways to remove the square bracket "\[" and "\]" in order to get a … --- ## [Unable to start logstash. The same configuration works in another node and index is created](https://discuss.elastic.co/t/unable-to-start-logstash-the-same-configuration-works-in-another-node-and-index-is-created/226519)
**Author:** [@ranjini](https://discuss.elastic.co/u/ranjini)\ **Replies:** 0\ **Last updated:** [April 4, 2020, 3:15pm UTC](https://discuss.elastic.co/t/unable-to-start-logstash-the-same-configuration-works-in-another-node-and-index-is-created/226519 "2020-04-04T15:15:30Z")
Logstash configuration works fine when output is tested with output { stdout { codec =\> rubydebug } } when sent to ES it errors out. \`input { file { path =\> "/opt/tomcat/logs/ds.log" start\_position =\> "beginning" … --- ## [My nested json is not getting parsed correctly, need help pls](https://discuss.elastic.co/t/my-nested-json-is-not-getting-parsed-correctly-need-help-pls/226317)
**Author:** [@Blason](https://discuss.elastic.co/u/Blason)\ **Replies:** 3\ **Last updated:** [April 4, 2020, 1:58pm UTC](https://discuss.elastic.co/t/my-nested-json-is-not-getting-parsed-correctly-need-help-pls/226317 "2020-04-04T13:58:13Z")
Hi Folks, Pertaining to my other thread I am trying to parse my below message with logstash however this is somehow is not happening. Can someone pls help me sorting out the issue pls? Or Any hint is really appreciated… --- ## [Logstash input multiline codec Entire file is getting merged into single line](https://discuss.elastic.co/t/logstash-input-multiline-codec-entire-file-is-getting-merged-into-single-line/226453)
**Author:** [@ranjith](https://discuss.elastic.co/u/ranjith)\ **Replies:** 1\ **Last updated:** [April 3, 2020, 8:40pm UTC](https://discuss.elastic.co/t/logstash-input-multiline-codec-entire-file-is-getting-merged-into-single-line/226453 "2020-04-03T20:40:55Z")
Hi All, I am creating a logstash pipeline to process the following logs My logstash config is input{ file{ path =\> "E:/Grok\_Practice/Practice/e.log" start\_position =\> "beginning" sincedb\_path =\> "LOG\_NULL" co… --- ## [Reading Config Files in pipeline.yml from another location](https://discuss.elastic.co/t/reading-config-files-in-pipeline-yml-from-another-location/225511)
**Author:** [@InfiniteLoops](https://discuss.elastic.co/u/InfiniteLoops)\ **Replies:** 2\ **Last updated:** [April 3, 2020, 8:37pm UTC](https://discuss.elastic.co/t/reading-config-files-in-pipeline-yml-from-another-location/225511 "2020-04-03T20:37:40Z")
I have all of my setup working fine. One issue is around the config files for the filters. Logstash is sitting on a remote machine and when a particular part of a filter needs to be changed this needs to be logged into a… --- ## [One logstash http input with multiple basic auth users](https://discuss.elastic.co/t/one-logstash-http-input-with-multiple-basic-auth-users/226442)
**Author:** [@mindhunt3r](https://discuss.elastic.co/u/mindhunt3r)\ **Replies:** 1\ **Last updated:** [April 3, 2020, 8:36pm UTC](https://discuss.elastic.co/t/one-logstash-http-input-with-multiple-basic-auth-users/226442 "2020-04-03T20:36:41Z")
I want to setup a logstash pipeline with multiple users/passwords and use the different usernames to direct output to different elasticsearch indexes where each username has read/write access to the specific index so sam… --- ## [Ruby Script Adds a New Field to LS Data Record?](https://discuss.elastic.co/t/ruby-script-adds-a-new-field-to-ls-data-record/226431)
**Author:** [@redapplesonly](https://discuss.elastic.co/u/redapplesonly)\ **Replies:** 4\ **Last updated:** [April 3, 2020, 5:07pm UTC](https://discuss.elastic.co/t/ruby-script-adds-a-new-field-to-ls-data-record/226431 "2020-04-03T17:07:44Z")
Hi Logstash Jedi Masters :slight\_smile: I need an example of a Ruby script which: Accepts a data record from Logstash Adds a new field to the record (e.g. : \[ “NewField” \] ) Fills in a default value (e.g. : \[ “NewFi… --- ## [Logstash 6.8 Input File not using exit\_after\_read](https://discuss.elastic.co/t/logstash-6-8-input-file-not-using-exit-after-read/226420)
**Author:** [@seibelj](https://discuss.elastic.co/u/seibelj)\ **Replies:** 5\ **Last updated:** [April 3, 2020, 4:51pm UTC](https://discuss.elastic.co/t/logstash-6-8-input-file-not-using-exit-after-read/226420 "2020-04-03T16:51:44Z")
Working with a client that has ELK 6.8.1 Their system produces a JSON file every minute (obviously a lot of files). I've been successful getting them into ES but having an issue with the number open. I'd like it to close… --- ## [Find source code of index date in elasticsearch output plugin](https://discuss.elastic.co/t/find-source-code-of-index-date-in-elasticsearch-output-plugin/226394)
**Author:** [@mabr3](https://discuss.elastic.co/u/mabr3)\ **Replies:** 2\ **Last updated:** [April 3, 2020, 2:40pm UTC](https://discuss.elastic.co/t/find-source-code-of-index-date-in-elasticsearch-output-plugin/226394 "2020-04-03T14:40:04Z")
Hi, After some research I found the confirmation that when using the elasticsearch output plugin and using the date for the index name, it automatically uses the '@timestamp' field. I was also interested in finding in t… --- ## [My field \`input\` is modified, and some events are filtered (against my will ;-))](https://discuss.elastic.co/t/my-field-input-is-modified-and-some-events-are-filtered-against-my-will/225690)
**Author:** [@Axl](https://discuss.elastic.co/u/Axl)\ **Replies:** 1\ **Last updated:** [April 3, 2020, 2:23pm UTC](https://discuss.elastic.co/t/my-field-input-is-modified-and-some-events-are-filtered-against-my-will/225690 "2020-04-03T14:23:05Z")
(Edited on March 31 with example) I have a working Filebeat+ELK config used to process logs from Cowrie (a honeypot). Except some log entries of Cowrie do not show in the end in Kibana, as if they had been filtered (by… --- ## [Is it redundant to use file input plugin with persistent queue?](https://discuss.elastic.co/t/is-it-redundant-to-use-file-input-plugin-with-persistent-queue/226297)
**Author:** [@danielyahn](https://discuss.elastic.co/u/danielyahn)\ **Replies:** 2\ **Last updated:** [April 3, 2020, 2:04pm UTC](https://discuss.elastic.co/t/is-it-redundant-to-use-file-input-plugin-with-persistent-queue/226297 "2020-04-03T14:04:24Z")
is it redundant to use file input plugin with persistent queue? Does persistent queue provide any additional delivery guarantee? Let's say file input plugin uses read mode sinceDB --- ## [How can i make logstash to read file with same filename after processing](https://discuss.elastic.co/t/how-can-i-make-logstash-to-read-file-with-same-filename-after-processing/226187)
**Author:** [@yigene](https://discuss.elastic.co/u/yigene)\ **Replies:** 3\ **Last updated:** [April 3, 2020, 2:00pm UTC](https://discuss.elastic.co/t/how-can-i-make-logstash-to-read-file-with-same-filename-after-processing/226187 "2020-04-03T14:00:49Z")
Hi, I have tried many ways but still not able to get it work basically, i have logstash to watching one folder and load into ES with read model input { file { path =\> \[ "/tmp/\*csv" \] type =\> "stp" tags =\> \["aa"\] … --- ## [Divide a field by a number in ruby](https://discuss.elastic.co/t/divide-a-field-by-a-number-in-ruby/226306)
**Author:** [@lfuentes](https://discuss.elastic.co/u/lfuentes)\ **Replies:** 1\ **Last updated:** [April 3, 2020, 1:45pm UTC](https://discuss.elastic.co/t/divide-a-field-by-a-number-in-ruby/226306 "2020-04-03T13:45:25Z")
Good, I need help please, I need to divide a number by 1000 using the ruby filter, could you help me with the code of that? I am really a beginner, I have something like this: ruby { code =\> "event.set('timetaken', eve… --- ## [Initial remote and local Filebeat config for logstash](https://discuss.elastic.co/t/initial-remote-and-local-filebeat-config-for-logstash/226263)
**Author:** [@Engineer33](https://discuss.elastic.co/u/Engineer33)\ **Replies:** 0\ **Last updated:** [April 2, 2020, 5:55pm UTC](https://discuss.elastic.co/t/initial-remote-and-local-filebeat-config-for-logstash/226263 "2020-04-02T17:55:16Z")
Hi all, Ive been in the filebeat.yml file and tried to do the initial config myself. I'm pretty new to the ELK stack and it processes but i made sure all ports were open but it still doesn't seem to connect. Below is … --- ## [Surprising error!](https://discuss.elastic.co/t/surprising-error/225582)
**Author:** [@GuyMuller](https://discuss.elastic.co/u/GuyMuller)\ **Replies:** 10\ **Last updated:** [April 3, 2020, 12:38pm UTC](https://discuss.elastic.co/t/surprising-error/225582 "2020-04-03T12:38:57Z")
Hello I have a problem with the creation of the config file and when launching the command. I have this error "do was unexpected." =\> do était inattendu. Command line : C:\\Program Files\\Elastic\\logstash\\bin\\logstash… --- ## [Logstash multiple jdbc plugin ingestion](https://discuss.elastic.co/t/logstash-multiple-jdbc-plugin-ingestion/226391)
**Author:** [@Vladpov](https://discuss.elastic.co/u/Vladpov)\ **Replies:** 0\ **Last updated:** [April 3, 2020, 10:57am UTC](https://discuss.elastic.co/t/logstash-multiple-jdbc-plugin-ingestion/226391 "2020-04-03T10:57:28Z")
Hey guys, I have set up MariaDB database from which I transport data to Elasticsearch via Logstash jdbc plugin. The ingestion of the data from database processlist table is being done every second. So every second I got… --- ## [Split data in message field to separate fields to display in kibana](https://discuss.elastic.co/t/split-data-in-message-field-to-separate-fields-to-display-in-kibana/226382)
**Author:** [@avinash221988](https://discuss.elastic.co/u/avinash221988)\ **Replies:** 0\ **Last updated:** [April 3, 2020, 10:05am UTC](https://discuss.elastic.co/t/split-data-in-message-field-to-separate-fields-to-display-in-kibana/226382 "2020-04-03T10:05:48Z")
Hi, I am using logstash and ES to feed data into kibana. Sending Json data to kibana.Below is the json data: {"Qmgrs":\[{"QueueManager":"QM1","Status":"Running"},{"QueueManager":"QM2","Status":"Ended immediately"},{"Que… --- ## [Tracing a log from Filebeat to Kibana](https://discuss.elastic.co/t/tracing-a-log-from-filebeat-to-kibana/226375)
**Author:** [@Axl](https://discuss.elastic.co/u/Axl)\ **Replies:** 0\ **Last updated:** [April 3, 2020, 9:49am UTC](https://discuss.elastic.co/t/tracing-a-log-from-filebeat-to-kibana/226375 "2020-04-03T09:49:45Z")
Hi, Is there a way to see the log entries at all levels: in my case Filebeat, then Logstash, then Elasticsearch, then Kibana? Indeed, I have some logs that I see in Filebeat, and logstash, which "disappear" afterwards :… --- ## [Logstash output Elasticsearch - mapper\_parsing\_exception](https://discuss.elastic.co/t/logstash-output-elasticsearch-mapper-parsing-exception/226256)
**Author:** [@abrejuin](https://discuss.elastic.co/u/abrejuin)\ **Replies:** 1\ **Last updated:** [April 3, 2020, 9:41am UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-mapper-parsing-exception/226256 "2020-04-03T09:41:04Z")
Hello, I upgraded Elastic from 5.3 to 6.0, I also upgraded Logstash from 5.6 to 6.8. Since then I have error messages on Logstash while trying to send logs to Elastic : \[WARN \] 2020-04-02 15:01:03.754 \[\[main\]\>worker0\]… --- ## [IMPORT .mib to Logstash snmptrap input](https://discuss.elastic.co/t/import-mib-to-logstash-snmptrap-input/226371)
**Author:** [@Diego\_Gutierrez](https://discuss.elastic.co/u/Diego_Gutierrez)\ **Replies:** 0\ **Last updated:** [April 3, 2020, 9:23am UTC](https://discuss.elastic.co/t/import-mib-to-logstash-snmptrap-input/226371 "2020-04-03T09:23:57Z")
Hi, I am having a lot of problem to import .mib files to logstash and transform it to .yaml file 'cause is the file that logstash can read. I've use the import.rb with ruby but I receive errors or directly I dont recei… --- ## [Read input from text file, use the same input in IF condition](https://discuss.elastic.co/t/read-input-from-text-file-use-the-same-input-in-if-condition/226362)
**Author:** [@Yogi\_1224](https://discuss.elastic.co/u/Yogi_1224)\ **Replies:** 0\ **Last updated:** [April 3, 2020, 7:58am UTC](https://discuss.elastic.co/t/read-input-from-text-file-use-the-same-input-in-if-condition/226362 "2020-04-03T07:58:03Z")
I want to read an input from text file and use the same input in IF condition to process some thing. Example: input =\> retrieve from file IF( input == "value" ) { Process Something......... } Can someone please hel… --- ## [Exporting CSV from Logstash](https://discuss.elastic.co/t/exporting-csv-from-logstash/226330)
**Author:** [@Sagar\_Mandal](https://discuss.elastic.co/u/Sagar_Mandal)\ **Replies:** 1\ **Last updated:** [April 3, 2020, 7:04am UTC](https://discuss.elastic.co/t/exporting-csv-from-logstash/226330 "2020-04-03T07:04:39Z")
Hi Team, How to Export CSV file from logstash ? is this even possible to export CSV file. If is there is any possible solution do help Thanks and Regards, Sagar Mandak --- ## [Get logstash to fail on undefined variable](https://discuss.elastic.co/t/get-logstash-to-fail-on-undefined-variable/223320)
**Author:** [@macau](https://discuss.elastic.co/u/macau)\ **Replies:** 1\ **Last updated:** [March 12, 2020, 11:34am UTC](https://discuss.elastic.co/t/get-logstash-to-fail-on-undefined-variable/223320 "2020-03-12T11:34:14Z")
output { file { path =\> "/srv/logstash-logs/%{somevar}.log" ... How can I get logstash to fail when an undefined variable is encountered? --- ## [Jdbc error](https://discuss.elastic.co/t/jdbc-error/226335)
**Author:** [@chelseia](https://discuss.elastic.co/u/chelseia)\ **Replies:** 0\ **Last updated:** [April 3, 2020, 5:52am UTC](https://discuss.elastic.co/t/jdbc-error/226335 "2020-04-03T05:52:22Z")
i am receiving the error ORA-01008: not all variables bound when i tried to run my logstash configuration file.I dont have any idea about this error.anyone can explain. input{ jdbc{ clean\_run =\> true jdbc\_driver\_lib… --- ## [Remove Subfield using ruby](https://discuss.elastic.co/t/remove-subfield-using-ruby/225055)
**Author:** [@inhinyera16](https://discuss.elastic.co/u/inhinyera16)\ **Replies:** 5\ **Last updated:** [April 2, 2020, 4:09pm UTC](https://discuss.elastic.co/t/remove-subfield-using-ruby/225055 "2020-04-02T16:09:55Z")
Anyone knows how to remove the attachment.documentcharacteristic field? using logstash ruby. Here the json data --- ## [Logstash gives error "Error: Address already in use" when ingesting filebeat with two inputs](https://discuss.elastic.co/t/logstash-gives-error-error-address-already-in-use-when-ingesting-filebeat-with-two-inputs/226168)
**Author:** [@chintanp](https://discuss.elastic.co/u/chintanp)\ **Replies:** 0\ **Last updated:** [April 2, 2020, 7:27am UTC](https://discuss.elastic.co/t/logstash-gives-error-error-address-already-in-use-when-ingesting-filebeat-with-two-inputs/226168 "2020-04-02T07:27:29Z")
Hi, I saw several similar posts, but none seemed to match my case. I got the open-source ELK stack from Bitnami and deployed it on AWS (Debian Linux 4.9). I am able to read the apache server logs locally, all the way t… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=378) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=380)