# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=573 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 574 --- ## [CSV as input and CSV as output](https://discuss.elastic.co/t/csv-as-input-and-csv-as-output/175160)
**Author:** [@mmk1995](https://discuss.elastic.co/u/mmk1995)\ **Replies:** 1\ **Last updated:** [April 3, 2019, 12:53pm UTC](https://discuss.elastic.co/t/csv-as-input-and-csv-as-output/175160 "2019-04-03T12:53:56Z")
Dear All, I am new to logstash and I am trying to input a .csv file to logstash and output it also as a .csv format for my testing. However, I find that the output .csv is not ordered row by row as the input .csv file.… --- ## [Received an event that has a different character encoding than you configured:)](https://discuss.elastic.co/t/received-an-event-that-has-a-different-character-encoding-than-you-configured/173513)
**Author:** [@nejmeddine\_ammar](https://discuss.elastic.co/u/nejmeddine_ammar)\ **Replies:** 2\ **Last updated:** [April 3, 2019, 12:47pm UTC](https://discuss.elastic.co/t/received-an-event-that-has-a-different-character-encoding-than-you-configured/173513 "2019-04-03T12:47:09Z")
when using logstash I see an error like this one : \[2019-03-14T12:59:39,516\]\[WARN \]\[logstash.codecs.plain \] Received an event that has a different character encoding than you configured. { :text=\>"08/02/2019 22:45:0… --- ## [Unable to match @timestamp to field created after creating dynamic mapping](https://discuss.elastic.co/t/unable-to-match-timestamp-to-field-created-after-creating-dynamic-mapping/174912)
**Author:** [@shrikantgulia](https://discuss.elastic.co/u/shrikantgulia)\ **Replies:** 16\ **Last updated:** [April 3, 2019, 11:07am UTC](https://discuss.elastic.co/t/unable-to-match-timestamp-to-field-created-after-creating-dynamic-mapping/174912 "2019-04-03T11:07:55Z")
Hello, I have created dynamic mapping {"doc": { "date\_detection": true, "dynamic\_date\_formats": "yyyy-MM-dd HH:mm:ss", "properties": { "created": { "type": "date", "for… --- ## [Cannot load mongodb output plugin](https://discuss.elastic.co/t/cannot-load-mongodb-output-plugin/175155)
**Author:** [@winkelwagen](https://discuss.elastic.co/u/winkelwagen)\ **Replies:** 0\ **Last updated:** [April 3, 2019, 10:06am UTC](https://discuss.elastic.co/t/cannot-load-mongodb-output-plugin/175155 "2019-04-03T10:06:55Z")
Hi, I'm trying to use the mongodb output plugin. The machine hosts Ubuntu 18.04, and logstash 6.5.4-1 is installed with apt. logstash then runs under the logstash user via systemd. The plugin is installed with /usr/… --- ## [Centralised pipeline management : separate filter + output pipeline](https://discuss.elastic.co/t/centralised-pipeline-management-separate-filter-output-pipeline/174176)
**Author:** [@annk](https://discuss.elastic.co/u/annk)\ **Replies:** 1\ **Last updated:** [April 3, 2019, 10:31am UTC](https://discuss.elastic.co/t/centralised-pipeline-management-separate-filter-output-pipeline/174176 "2019-04-03T10:31:43Z")
Hi , We have a tcp input plugin listening on a port. I have few applications pushing logs to this pipeline .I am just wondering if its possible to create another pipeline with just filter and output section. My pipelin… --- ## [Events to Exec output plugin](https://discuss.elastic.co/t/events-to-exec-output-plugin/175142)
**Author:** [@artobstrel](https://discuss.elastic.co/u/artobstrel)\ **Replies:** 0\ **Last updated:** [April 3, 2019, 8:49am UTC](https://discuss.elastic.co/t/events-to-exec-output-plugin/175142 "2019-04-03T08:49:23Z")
Hello guys! I'll try send event to my program on python with exec pluginoutput { exec { command =\> "python /somepath/my\_script.py" } } I can send an arbitrary fieldcommand =\> "pytho… --- ## [Extracting specific lines of a text using ruby](https://discuss.elastic.co/t/extracting-specific-lines-of-a-text-using-ruby/175134)
**Author:** [@yasin\_mohammed](https://discuss.elastic.co/u/yasin_mohammed)\ **Replies:** 1\ **Last updated:** [April 3, 2019, 9:47am UTC](https://discuss.elastic.co/t/extracting-specific-lines-of-a-text-using-ruby/175134 "2019-04-03T09:47:51Z")
I am trying to extract only the lines 'My' from the message field.I am using a ruby code . The code runs fine when I am using it in a Ruby console but when I send it through logstash it does not extract the specific line … --- ## [Dead\_letter\_queue](https://discuss.elastic.co/t/dead-letter-queue/174063)
**Author:** [@xavi23](https://discuss.elastic.co/u/xavi23)\ **Replies:** 6\ **Last updated:** [April 3, 2019, 7:18am UTC](https://discuss.elastic.co/t/dead-letter-queue/174063 "2019-04-03T07:18:48Z")
Hello, Actually I'm using a cluster of Amazon ElasticSearch and I'm trying to load some documents from S3 (input) using logstash. I've one pipeline (file.conf) with 25 inputs , one for each different folder in S3. The… --- ## [Logstash 6.7 and Java 11 with .deb or .rpm repositories not working](https://discuss.elastic.co/t/logstash-6-7-and-java-11-with-deb-or-rpm-repositories-not-working/174505)
**Author:** [@arviaja](https://discuss.elastic.co/u/arviaja)\ **Replies:** 3\ **Last updated:** [April 3, 2019, 6:58am UTC](https://discuss.elastic.co/t/logstash-6-7-and-java-11-with-deb-or-rpm-repositories-not-working/174505 "2019-04-03T06:58:45Z")
Hi there, I've been trying to install Logstash 6.7 from the repositories which Elastic provides (based on these guides RPM and DEB and Java 11 from Oracle and alternatively with OpenJDK 11. I've tried both on Ubuntu 18… --- ## [Syslog input message cutting](https://discuss.elastic.co/t/syslog-input-message-cutting/175105)
**Author:** [@bodi4](https://discuss.elastic.co/u/bodi4)\ **Replies:** 0\ **Last updated:** [April 3, 2019, 6:48am UTC](https://discuss.elastic.co/t/syslog-input-message-cutting/175105 "2019-04-03T06:48:49Z")
Hi, Having issue with syslog input. Sending syslog events from F5 to logstash with http payload data. Payload field can be with xml data http\_payload="\... Logstash parsing syslo… --- ## [Issue using logstash-input-jdbc to load mongo data](https://discuss.elastic.co/t/issue-using-logstash-input-jdbc-to-load-mongo-data/174914)
**Author:** [@supritha](https://discuss.elastic.co/u/supritha)\ **Replies:** 2\ **Last updated:** [April 3, 2019, 4:09am UTC](https://discuss.elastic.co/t/issue-using-logstash-input-jdbc-to-load-mongo-data/174914 "2019-04-03T04:09:01Z")
Hello, I'm using logstash to load MongoDB data to elasticsearch. I get MongoDB data inside the map field (inside source). I want data to be part of the source, not in a separate field. I used db.collection.find() statem… --- ## [Detected ambiguous Field Reference. How to fix it exactly?](https://discuss.elastic.co/t/detected-ambiguous-field-reference-how-to-fix-it-exactly/175076)
**Author:** [@jcaballero](https://discuss.elastic.co/u/jcaballero)\ **Replies:** 2\ **Last updated:** [April 2, 2019, 9:33pm UTC](https://discuss.elastic.co/t/detected-ambiguous-field-reference-how-to-fix-it-exactly/175076 "2019-04-02T21:33:55Z")
Hi LogStash 6.6.2 I see several topics with similar questions, but answers are unclear to me. So I ask again. In my filters, I have this mutate { add\_field =\> { "executionhost" =\> "%{host\[name\]}" } } mutate { sp… --- ## [WinBeat Config for AWS](https://discuss.elastic.co/t/winbeat-config-for-aws/175070)
**Author:** [@Ericrs22](https://discuss.elastic.co/u/Ericrs22)\ **Replies:** 3\ **Last updated:** [April 2, 2019, 9:04pm UTC](https://discuss.elastic.co/t/winbeat-config-for-aws/175070 "2019-04-02T21:04:13Z")
Has anyone set this up? I'm trying to follow the readme and other configs I've seen however I keep getting errors connecting to it. 2019-04-02T13:18:18.271-0700 ERROR pipeline/output.go:100 Failed to connect to backoff(… --- ## [Logstash columns format](https://discuss.elastic.co/t/logstash-columns-format/175065)
**Author:** [@He\_Pamela](https://discuss.elastic.co/u/He_Pamela)\ **Replies:** 1\ **Last updated:** [April 2, 2019, 8:39pm UTC](https://discuss.elastic.co/t/logstash-columns-format/175065 "2019-04-02T20:39:11Z")
Hi, I have this issue. I'm trying to send a CSV file to Elasticsearch using logstash. This is the log I get: \[2019-04-02T14:22:18,470\]\[WARN \]\[logstash.filters.csv \] Error parsing csv {:field=>"message", :source=>"… --- ## [Logstash update only 1 field](https://discuss.elastic.co/t/logstash-update-only-1-field/175066)
**Author:** [@R\_H1](https://discuss.elastic.co/u/R_H1)\ **Replies:** 0\ **Last updated:** [April 2, 2019, 7:58pm UTC](https://discuss.elastic.co/t/logstash-update-only-1-field/175066 "2019-04-02T19:58:23Z")
Hi, I'm parsing a logfile and each line has in a column "Action" either "CREATE" or "CLEAR". If the Action field contains CREATE, I want to create a new document as normal. If the Action contains CLEAR it just want to u… --- ## [Logstash syslog output message =\> %{\_source}](https://discuss.elastic.co/t/logstash-syslog-output-message-source/175028)
**Author:** [@souf\_el\_badraoui](https://discuss.elastic.co/u/souf_el_badraoui)\ **Replies:** 1\ **Last updated:** [April 2, 2019, 5:47pm UTC](https://discuss.elastic.co/t/logstash-syslog-output-message-source/175028 "2019-04-02T17:47:40Z")
Hello, We are trying to forward windows event log shipped with winlogbeat via logstash output syslog. The problem is that we want to forward all fields (ideally key1=value1 key2=value2...) and not just the message field… --- ## [How to start Grok Pattern](https://discuss.elastic.co/t/how-to-start-grok-pattern/174556)
**Author:** [@nagr](https://discuss.elastic.co/u/nagr)\ **Replies:** 3\ **Last updated:** [April 2, 2019, 5:10pm UTC](https://discuss.elastic.co/t/how-to-start-grok-pattern/174556 "2019-04-02T17:10:42Z")
Hi All, Could someone help me to start the log analyze using grok filter for logstash method, I am just trying to capture the log file from windows machine below path "C:\\Windows\\Logs\\CBS\\CBS.log" from this line I ju… --- ## [Guidance With S3 Input Plugin](https://discuss.elastic.co/t/guidance-with-s3-input-plugin/175005)
**Author:** [@pcarcel](https://discuss.elastic.co/u/pcarcel)\ **Replies:** 2\ **Last updated:** [April 2, 2019, 3:55pm UTC](https://discuss.elastic.co/t/guidance-with-s3-input-plugin/175005 "2019-04-02T15:55:56Z")
I am using the S3 input plugin on a project I am working on. Each day a couple hundred thousand files are sent to different buckets, then processed by logstash. I am using the options to backup the data to the same buc… --- ## [Enable logstash to read on port number](https://discuss.elastic.co/t/enable-logstash-to-read-on-port-number/174934)
**Author:** [@mouli\_v](https://discuss.elastic.co/u/mouli_v)\ **Replies:** 7\ **Last updated:** [April 2, 2019, 2:50pm UTC](https://discuss.elastic.co/t/enable-logstash-to-read-on-port-number/174934 "2019-04-02T14:50:50Z")
Hi There, I need some help in testing this. Below is the logstash config file\[logstash-6.5.4/config/logstash-sample-4.conf\]. I would like to feed the logstash with elasticsearch logs, which runs on port 9200. input { … --- ## [Why Logstash is receving data of past time?](https://discuss.elastic.co/t/why-logstash-is-receving-data-of-past-time/175012)
**Author:** [@Deepak2](https://discuss.elastic.co/u/Deepak2)\ **Replies:** 1\ **Last updated:** [April 2, 2019, 2:41pm UTC](https://discuss.elastic.co/t/why-logstash-is-receving-data-of-past-time/175012 "2019-04-02T14:41:55Z")
Hi, When i am running logstash, then i noticed that it is receving data from filebeat of long time back (means if a am running at 7 pm then data of 9 am is recevied.) Also logstash is receving data from filebeat wh… --- ## [Jdbc\_properties under sequel\_opts in jdbc plugin logstash](https://discuss.elastic.co/t/jdbc-properties-under-sequel-opts-in-jdbc-plugin-logstash/174962)
**Author:** [@rameshkv1989](https://discuss.elastic.co/u/rameshkv1989)\ **Replies:** 1\ **Last updated:** [April 2, 2019, 2:20pm UTC](https://discuss.elastic.co/t/jdbc-properties-under-sequel-opts-in-jdbc-plugin-logstash/174962 "2019-04-02T14:20:45Z")
I need to pass two parameters "encryptionAlgorithm" and "securityMechanism" under jdbc\_properties , can anyone please advise if poosible. Below thing is not working jdbc { sequel\_opts =\> { jdbc\_properties =\> {"securit… --- ## [USe Event API from ruby method](https://discuss.elastic.co/t/use-event-api-from-ruby-method/174993)
**Author:** [@ep4sh](https://discuss.elastic.co/u/ep4sh)\ **Replies:** 5\ **Last updated:** [April 2, 2019, 2:11pm UTC](https://discuss.elastic.co/t/use-event-api-from-ruby-method/174993 "2019-04-02T14:11:36Z")
Trying to do tagging inside my own method. def jP(str) begin parsed = JSON.parse(str) rescue =\> event.tag("CATCH") return end end But got an error: Ruby exception occurred: u… --- ## [Unable to parse CSV input from Filebeat -\> Logstash](https://discuss.elastic.co/t/unable-to-parse-csv-input-from-filebeat-logstash/174952)
**Author:** [@avj1986](https://discuss.elastic.co/u/avj1986)\ **Replies:** 2\ **Last updated:** [April 2, 2019, 1:37pm UTC](https://discuss.elastic.co/t/unable-to-parse-csv-input-from-filebeat-logstash/174952 "2019-04-02T13:37:06Z")
Hi, I am trying to read a CSV file using Filebeat and sending the data to Logstash. On Logstash, I need to parse this CSV data and load it into ElasticSearch. I am able to ship the data from Filebeat to Logstash to El… --- ## [Logstash neither detect the newly added data in CSV file nor push it into elasticsearch](https://discuss.elastic.co/t/logstash-neither-detect-the-newly-added-data-in-csv-file-nor-push-it-into-elasticsearch/174938)
**Author:** [@cheriemilk](https://discuss.elastic.co/u/cheriemilk)\ **Replies:** 2\ **Last updated:** [April 2, 2019, 1:21pm UTC](https://discuss.elastic.co/t/logstash-neither-detect-the-newly-added-data-in-csv-file-nor-push-it-into-elasticsearch/174938 "2019-04-02T13:21:30Z")
Hi team, I have a CSV file which has 3 records at the beginning and parsed by logstash successfully. (I didn't close logstash at this moment) Later, I open the file and add forth records and want logstash detect the ne… --- ## [Error while importing CSV file with Logstash](https://discuss.elastic.co/t/error-while-importing-csv-file-with-logstash/174834)
**Author:** [@matheusribeirogarcia](https://discuss.elastic.co/u/matheusribeirogarcia)\ **Replies:** 4\ **Last updated:** [April 2, 2019, 12:52pm UTC](https://discuss.elastic.co/t/error-while-importing-csv-file-with-logstash/174834 "2019-04-02T12:52:36Z")
While it seems like my config file is right, it still doesn't run properly, accusing it of missing a # I'm executing this command: logstash -f C:\\Users\\CONSISTE\\Desktop\\logstash\\dados\\teste.config message=\>"Expected … --- ## [How to make Logstash multiple output?](https://discuss.elastic.co/t/how-to-make-logstash-multiple-output/174985)
**Author:** [@allfreed](https://discuss.elastic.co/u/allfreed)\ **Replies:** 1\ **Last updated:** [April 2, 2019, 12:51pm UTC](https://discuss.elastic.co/t/how-to-make-logstash-multiple-output/174985 "2019-04-02T12:51:02Z")
Hello , i want to make multiple output , for example elasticsearch and kafka. I was created output.conf output { elasticsearch { hosts =\> "ip:9200" index =\> "ubuntu18" } kafka { bootstrap\_server =\> "ip:9092" co… --- ## [Translate plugin using csv](https://discuss.elastic.co/t/translate-plugin-using-csv/174762)
**Author:** [@Mickael\_Humphreys](https://discuss.elastic.co/u/Mickael_Humphreys)\ **Replies:** 2\ **Last updated:** [April 2, 2019, 10:40am UTC](https://discuss.elastic.co/t/translate-plugin-using-csv/174762 "2019-04-02T10:40:12Z")
Hello, I'm trying to parse all the columns of a CSV file (except the first one, obviously). The plugin only get the second column as result of the filter. All the other column are ignored. It should be possible, acco… --- ## [Logstash stops sending docs to elasticsearch, starts over after restart of process](https://discuss.elastic.co/t/logstash-stops-sending-docs-to-elasticsearch-starts-over-after-restart-of-process/174659)
**Author:** [@yossib](https://discuss.elastic.co/u/yossib)\ **Replies:** 3\ **Last updated:** [April 2, 2019, 8:14am UTC](https://discuss.elastic.co/t/logstash-stops-sending-docs-to-elasticsearch-starts-over-after-restart-of-process/174659 "2019-04-02T08:14:48Z")
elasticsearch is running on a 3 node cluster, logstash and filebeat are running on a single node all version are: 6.5.1 logs are being sent to elasticsearch for short periods (~2 hours), then they stop... only after re… --- ## [JDBC Filter error when running as service](https://discuss.elastic.co/t/jdbc-filter-error-when-running-as-service/172985)
**Author:** [@hubertn](https://discuss.elastic.co/u/hubertn)\ **Replies:** 1\ **Last updated:** [April 2, 2019, 6:43am UTC](https://discuss.elastic.co/t/jdbc-filter-error-when-running-as-service/172985 "2019-04-02T06:43:02Z")
We have a logstash's pipeline that use JDBC filter. When we ran the pipeline as service using: systemctl start our.service. We have the following error; but when we ran as command everything looks fine. Does anyone … --- ## [Logstash - Framing grok filter plugin](https://discuss.elastic.co/t/logstash-framing-grok-filter-plugin/174910)
**Author:** [@Gomathi\_Meena](https://discuss.elastic.co/u/Gomathi_Meena)\ **Replies:** 0\ **Last updated:** [April 2, 2019, 5:26am UTC](https://discuss.elastic.co/t/logstash-framing-grok-filter-plugin/174910 "2019-04-02T05:26:50Z")
Hi , I have a bunch of students records in my database(MySQL) and i want to pull those to elastic search which matches my pattern . Now i need to write a configuration file for this .I'm clear about writing input and ou… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=572) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=574)