# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=307 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 308 --- ## [Logstash and CSV - append to field while upsert](https://discuss.elastic.co/t/logstash-and-csv-append-to-field-while-upsert/247071)
**Author:** [@AnthonyA](https://discuss.elastic.co/u/AnthonyA)\ **Replies:** 3\ **Last updated:** [September 4, 2020, 7:47am UTC](https://discuss.elastic.co/t/logstash-and-csv-append-to-field-while-upsert/247071 "2020-09-04T07:47:10Z")
Hi there :grinning: I'm using logstash. My input is a CSV file. I'm using one of the columns as the document\_id. There are lines in the CSV file that have the same document\_id. In my output, I'm using upsert. When Logs… --- ## [Resolved - Logstash - CSV parse failure field thai language](https://discuss.elastic.co/t/resolved-logstash-csv-parse-failure-field-thai-language/247478)
**Author:** [@111207](https://discuss.elastic.co/u/111207)\ **Replies:** 0\ **Last updated:** [September 4, 2020, 4:25am UTC](https://discuss.elastic.co/t/resolved-logstash-csv-parse-failure-field-thai-language/247478 "2020-09-04T04:25:51Z")
This is my csv field example input 50001979,60004305,DHL Reseller Nakorn In,THLZ3EL29,01-08-20 8:37,01-08-20 19:07,ศรีสุดา เตียเจริญ,34/53 ม.4 แขวง/ลาดพร้าว,Lat Phrao,Bangkok,10230,897957585,นภวรรณ สวัสดิ์มงคลกุล,504/1 … --- ## [Grok WinEvt log](https://discuss.elastic.co/t/grok-winevt-log/247466)
**Author:** [@Jose\_Campos](https://discuss.elastic.co/u/Jose_Campos)\ **Replies:** 0\ **Last updated:** [September 4, 2020, 1:33am UTC](https://discuss.elastic.co/t/grok-winevt-log/247466 "2020-09-04T01:33:10Z")
Hello I was trying to grok for this CSV line that I downloaded from my windows 10 logs, I see that it is separated by commas, but the last value comes lines like this, does anyone have any ideas? 8/26/2020,5:34:23 PM,S… --- ## [Entries Duplication](https://discuss.elastic.co/t/entries-duplication/247448)
**Author:** [@js7](https://discuss.elastic.co/u/js7)\ **Replies:** 3\ **Last updated:** [September 3, 2020, 11:18pm UTC](https://discuss.elastic.co/t/entries-duplication/247448 "2020-09-03T23:18:33Z")
Hi, when I have 2 pipelines active at the same time, the data from one is somehow duplicated/redirected to the other pipeline, when I'm not using pipeline-to-pipeline communication. The output host of both are the same… --- ## [Create arrays based on certain text in a field](https://discuss.elastic.co/t/create-arrays-based-on-certain-text-in-a-field/246811)
**Author:** [@kopacko](https://discuss.elastic.co/u/kopacko)\ **Replies:** 11\ **Last updated:** [September 3, 2020, 4:29pm UTC](https://discuss.elastic.co/t/create-arrays-based-on-certain-text-in-a-field/246811 "2020-09-03T16:29:51Z")
I am parsing chat messages and I want to create three arrays for tag cloud purposes. One for normal chat words, one for user tags, and one for chat emotes. For example, take this chat message: WE LOVE YOU @CHANDLER, Y… --- ## [How to handle rotated logs](https://discuss.elastic.co/t/how-to-handle-rotated-logs/247341)
**Author:** [@msk\_76](https://discuss.elastic.co/u/msk_76)\ **Replies:** 3\ **Last updated:** [September 3, 2020, 4:07pm UTC](https://discuss.elastic.co/t/how-to-handle-rotated-logs/247341 "2020-09-03T16:07:38Z")
I am analyzing logs using logstash that are getting updated realtime from an application. The application has an inbuilt log rotation process like this. My application always writes to fsl.stream and once this fsl.stream… --- ## [Clarification towards logstash batch\_size and memory usage](https://discuss.elastic.co/t/clarification-towards-logstash-batch-size-and-memory-usage/247073)
**Author:** [@vaishno.avi](https://discuss.elastic.co/u/vaishno.avi)\ **Replies:** 4\ **Last updated:** [September 3, 2020, 3:37pm UTC](https://discuss.elastic.co/t/clarification-towards-logstash-batch-size-and-memory-usage/247073 "2020-09-03T15:37:02Z")
Hello All, I have two concerns I have a pipeline.yml where I am explicitly setting batch size as 100, but when I start logstash it shows me 125, is this an expected behavior ? config - pipeline.id: filebeat path.c… --- ## [Explain Periodic\_flush in logsatsh common fields](https://discuss.elastic.co/t/explain-periodic-flush-in-logsatsh-common-fields/247328)
**Author:** [@SKumarMN](https://discuss.elastic.co/u/SKumarMN)\ **Replies:** 1\ **Last updated:** [September 3, 2020, 3:35pm UTC](https://discuss.elastic.co/t/explain-periodic-flush-in-logsatsh-common-fields/247328 "2020-09-03T15:35:29Z")
Hi, Under Logstash common fields, I see a field names Periodic\_flush. Could you please explain whats is used for and also give an example with respect to translate plugin https://www.elastic.co/guide/en/logstash/curren… --- ## [Logstash WMI Input Plugin donot work](https://discuss.elastic.co/t/logstash-wmi-input-plugin-donot-work/247404)
**Author:** [@souravtyson](https://discuss.elastic.co/u/souravtyson)\ **Replies:** 0\ **Last updated:** [September 3, 2020, 2:26pm UTC](https://discuss.elastic.co/t/logstash-wmi-input-plugin-donot-work/247404 "2020-09-03T14:26:06Z")
I am new to logstash and i want to gather the performance metrics of the windows system. I have been trying to run the WMI input plugin of logstash in my local setup to gather data, but it is not getting executed .This i… --- ## [Rename fields using a dictionary](https://discuss.elastic.co/t/rename-fields-using-a-dictionary/246413)
**Author:** [@frank\_esg](https://discuss.elastic.co/u/frank_esg)\ **Replies:** 2\ **Last updated:** [September 3, 2020, 2:20pm UTC](https://discuss.elastic.co/t/rename-fields-using-a-dictionary/246413 "2020-09-03T14:20:05Z")
Hi Folks, is it possible in Logstash to rename fields (received as json) based on a dictionary File? Basically the same function like the translate filter does, but not for values but for field names. Background: At th… --- ## [Syslog source ip](https://discuss.elastic.co/t/syslog-source-ip/247363)
**Author:** [@mutt13y](https://discuss.elastic.co/u/mutt13y)\ **Replies:** 3\ **Last updated:** [September 3, 2020, 1:26pm UTC](https://discuss.elastic.co/t/syslog-source-ip/247363 "2020-09-03T13:26:40Z")
Hi, The syslog input plugin is creating a field based on the source IP of the syslog message. However for some hosts this is put into \[host\] and for other hosts it is going into \[host\]\[ip\] I cant figure out why this is… --- ## [Logstash - Simple Log Ingest](https://discuss.elastic.co/t/logstash-simple-log-ingest/247383)
**Author:** [@slash665](https://discuss.elastic.co/u/slash665)\ **Replies:** 1\ **Last updated:** [September 3, 2020, 1:22pm UTC](https://discuss.elastic.co/t/logstash-simple-log-ingest/247383 "2020-09-03T13:22:12Z")
Hello there, I'm a total newb to ELK, so at this point I am just trying to ingest a log file from a specific path using a logstash configuration file in conf.d . it's just not working, however... This is what my conf f… --- ## [KV filter not splitting log](https://discuss.elastic.co/t/kv-filter-not-splitting-log/247380)
**Author:** [@varun1992](https://discuss.elastic.co/u/varun1992)\ **Replies:** 0\ **Last updated:** [September 3, 2020, 12:36pm UTC](https://discuss.elastic.co/t/kv-filter-not-splitting-log/247380 "2020-09-03T12:36:47Z")
My config file as below. It's not splitting log read by grok input { file { path =\> "/log/ForwardTrafficLog.log" start\_position =\> "beginning" } } filter { grok { match =\> { "message" =\> \["%{GR… --- ## [Logstash - grok - ruby and random numbers](https://discuss.elastic.co/t/logstash-grok-ruby-and-random-numbers/247367)
**Author:** [@BBQigniter](https://discuss.elastic.co/u/BBQigniter)\ **Replies:** 0\ **Last updated:** [September 3, 2020, 10:36am UTC](https://discuss.elastic.co/t/logstash-grok-ruby-and-random-numbers/247367 "2020-09-03T10:36:02Z")
hello, maybe someone has an idea here. we use following setup for our log-filtering: source -- (dns-rr) --\> 3 x logstash -\> 3-node-rabbitmq-cluster -\> 3 x logstash -\> elasticsearch-cluster on the 3-node rabbitmq-clus… --- ## [Logstash-input-wmi error in logs](https://discuss.elastic.co/t/logstash-input-wmi-error-in-logs/246796)
**Author:** [@Adriann](https://discuss.elastic.co/u/Adriann)\ **Replies:** 3\ **Last updated:** [September 3, 2020, 9:53am UTC](https://discuss.elastic.co/t/logstash-input-wmi-error-in-logs/246796 "2020-09-03T09:53:50Z")
Hello, I installed logstash-wmi plugin into my logstash installation 7.8 using logstash executable and command install. I am not able to move forward with my configuration becouse of "(LoadError) load error: win32ole/w… --- ## [Logstash not detecting newly added records in elastic](https://discuss.elastic.co/t/logstash-not-detecting-newly-added-records-in-elastic/247358)
**Author:** [@B\_K\_V\_GOPI\_KRISHNA\_G](https://discuss.elastic.co/u/B_K_V_GOPI_KRISHNA_G)\ **Replies:** 0\ **Last updated:** [September 3, 2020, 9:45am UTC](https://discuss.elastic.co/t/logstash-not-detecting-newly-added-records-in-elastic/247358 "2020-09-03T09:45:48Z")
I have a sql db and I have logstash running on it for every 1min with docupsert true , everything is fine . But when i add a record into elastic using my backend code , its being pushed into the elastic successfully bu… --- ## [Logstash Configuration Error](https://discuss.elastic.co/t/logstash-configuration-error/247211)
**Author:** [@Chandana\_Koppal](https://discuss.elastic.co/u/Chandana_Koppal)\ **Replies:** 2\ **Last updated:** [September 3, 2020, 6:28am UTC](https://discuss.elastic.co/t/logstash-configuration-error/247211 "2020-09-03T06:28:32Z")
\[2020-09-02T16:11:38,526\]\[ERROR\]\[logstash.agent \] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of \[ \\… --- ## [Tracking\_column not found in dataset when using JDBC driver with mongodb (input)](https://discuss.elastic.co/t/tracking-column-not-found-in-dataset-when-using-jdbc-driver-with-mongodb-input/246706)
**Author:** [@timgau](https://discuss.elastic.co/u/timgau)\ **Replies:** 9\ **Last updated:** [September 3, 2020, 4:50am UTC](https://discuss.elastic.co/t/tracking-column-not-found-in-dataset-when-using-jdbc-driver-with-mongodb-input/246706 "2020-09-03T04:50:24Z")
I'm quite unclear of how to use :sql\_last\_value in my mongodb query statement like: statement =\> "var lastId = ':sql\_last\_value'; db.docs.find({ \_id: { $gte : lastId }})" my config is like this input { jdbc { jd… --- ## [Is it possible to access cloned event to add some more fields to it later on](https://discuss.elastic.co/t/is-it-possible-to-access-cloned-event-to-add-some-more-fields-to-it-later-on/247313)
**Author:** [@pk.241011](https://discuss.elastic.co/u/pk.241011)\ **Replies:** 1\ **Last updated:** [September 3, 2020, 3:15am UTC](https://discuss.elastic.co/t/is-it-possible-to-access-cloned-event-to-add-some-more-fields-to-it-later-on/247313 "2020-09-03T03:15:19Z")
Hi All, I have cloned an event and removed a field from it. Now I am wondering if I can later on access this cloned event in the logstash and add another field to it. clone { clones =\> \["cloned"\] … --- ## [Scheduling Logstash in Windows](https://discuss.elastic.co/t/scheduling-logstash-in-windows/247317)
**Author:** [@whyptrap](https://discuss.elastic.co/u/whyptrap)\ **Replies:** 0\ **Last updated:** [September 3, 2020, 2:52am UTC](https://discuss.elastic.co/t/scheduling-logstash-in-windows/247317 "2020-09-03T02:52:57Z")
Hi, i think i need help. I have 2 logstash configuration files with different csv file on input, and output to different index pattern too. I've try to create 2 task scheduler to run each file, and stop the task after … --- ## [Logstash failed to run](https://discuss.elastic.co/t/logstash-failed-to-run/247294)
**Author:** [@billzhang1123](https://discuss.elastic.co/u/billzhang1123)\ **Replies:** 2\ **Last updated:** [September 3, 2020, 12:47am UTC](https://discuss.elastic.co/t/logstash-failed-to-run/247294 "2020-09-03T00:47:45Z")
I have this configuration below running against the latest release of logstash. but received the error below. any suggestion? input { file { path =\> "~/logstash-data/ActionTrackingGeneral.csv" start\_position =\> "begi… --- ## [Up to date way to ingest logs from Java application using log4j2](https://discuss.elastic.co/t/up-to-date-way-to-ingest-logs-from-java-application-using-log4j2/241300)
**Author:** [@Daniele\_Renda](https://discuss.elastic.co/u/Daniele_Renda)\ **Replies:** 4\ **Last updated:** [September 2, 2020, 11:07pm UTC](https://discuss.elastic.co/t/up-to-date-way-to-ingest-logs-from-java-application-using-log4j2/241300 "2020-09-02T23:07:02Z")
Hi, I've a Spring Boot 2.3.1 application that produces some logs. I'd like to ingest them into my ElasticSearch stack (I'm hosted on Elastic cloud). What's the better way to do that? I saw the logstash log4j plugin is d… --- ## [\_grokparsefailure Tag in all parsed logs with multiple grok filter](https://discuss.elastic.co/t/grokparsefailure-tag-in-all-parsed-logs-with-multiple-grok-filter/247200)
**Author:** [@mohammadamin75](https://discuss.elastic.co/u/mohammadamin75)\ **Replies:** 5\ **Last updated:** [September 2, 2020, 5:22pm UTC](https://discuss.elastic.co/t/grokparsefailure-tag-in-all-parsed-logs-with-multiple-grok-filter/247200 "2020-09-02T17:22:34Z")
I am trying to parse minecraft log with Elastic Stack and I'v faced a very strange problem (likely strange for me!) all line of my log get parsed correctly but I got \_grokparsefailure tag in everyone of them. my logsta… --- ## [Parsing nested json](https://discuss.elastic.co/t/parsing-nested-json/247228)
**Author:** [@sana1](https://discuss.elastic.co/u/sana1)\ **Replies:** 1\ **Last updated:** [September 2, 2020, 5:07pm UTC](https://discuss.elastic.co/t/parsing-nested-json/247228 "2020-09-02T17:07:05Z")
I need help in parsing a nested array data ''' "abc" =\> \[ \[0\] { "obj" =\> "78", "val" =\> "0/100", "name" =\> "xyx", "email" =\> "xyz@gmail.com" }, \[1\] { "obj" =\> "78", "val" =\> "0/100", "name" =\> "xyx", "email"… --- ## [Logstash: Elasticsearch input plugin doesn't work](https://discuss.elastic.co/t/logstash-elasticsearch-input-plugin-doesnt-work/247145)
**Author:** [@Igor\_Olikh](https://discuss.elastic.co/u/Igor_Olikh)\ **Replies:** 3\ **Last updated:** [September 2, 2020, 4:51pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-input-plugin-doesnt-work/247145 "2020-09-02T16:51:25Z")
It sounds strange (at least for me) that the output plugin works: elasticsearch { hosts =\> \["https://elastict:9200"\] user =\> "elastic" password =\> "PASSWORD" ssl =\> true ssl\_certificate\_verification =\> true cacer… --- ## [Logstash Cache when elasticsearch not reachable](https://discuss.elastic.co/t/logstash-cache-when-elasticsearch-not-reachable/247181)
**Author:** [@kkhadka](https://discuss.elastic.co/u/kkhadka)\ **Replies:** 3\ **Last updated:** [September 2, 2020, 4:46pm UTC](https://discuss.elastic.co/t/logstash-cache-when-elasticsearch-not-reachable/247181 "2020-09-02T16:46:35Z")
if the elasticsearch is not reachable and data is being sent to logstash, does that data be cache by logstash until the elasticsearch is reachable? --- ## [How to extract key-value pairs automatically from mixed type of message](https://discuss.elastic.co/t/how-to-extract-key-value-pairs-automatically-from-mixed-type-of-message/247264)
**Author:** [@kelk](https://discuss.elastic.co/u/kelk)\ **Replies:** 0\ **Last updated:** [September 2, 2020, 4:17pm UTC](https://discuss.elastic.co/t/how-to-extract-key-value-pairs-automatically-from-mixed-type-of-message/247264 "2020-09-02T16:17:25Z")
hi We have a mixed set of message coming from filebeat. The message is some form of "YAML/json" mix \[WARN \] 2020-09-02 16:03:55.966 \[\[pipline\_id\]\>worker6\] elasticsearch - Could not index event to Elasticsearch. {:statu… --- ## [Check the target object of a local\_lookup for jdbc\_static](https://discuss.elastic.co/t/check-the-target-object-of-a-local-lookup-for-jdbc-static/246975)
**Author:** [@chapmantrain](https://discuss.elastic.co/u/chapmantrain)\ **Replies:** 2\ **Last updated:** [September 2, 2020, 4:04pm UTC](https://discuss.elastic.co/t/check-the-target-object-of-a-local-lookup-for-jdbc-static/246975 "2020-09-02T16:04:10Z")
How would I check to see if the local\_lookup found the host name in the following conf snip. I tried an if statement just before the add\_fields but logstash did not like that. if \[device\] {...} = Expected one of \[ \\t\\r\\… --- ## [Output when have multiple logstash servers in cluster](https://discuss.elastic.co/t/output-when-have-multiple-logstash-servers-in-cluster/247251)
**Author:** [@nnovaes](https://discuss.elastic.co/u/nnovaes)\ **Replies:** 0\ **Last updated:** [September 2, 2020, 2:47pm UTC](https://discuss.elastic.co/t/output-when-have-multiple-logstash-servers-in-cluster/247251 "2020-09-02T14:47:18Z")
If i have, let's say, 3 servers in a logstash cluster, can i coordinate them somehow so when i configure syslog output only one of those servers will actually send the syslog message? otherwise all 3 would send and the r… --- ## [Can we send logs to webhook URL?](https://discuss.elastic.co/t/can-we-send-logs-to-webhook-url/247100)
**Author:** [@Aishvarya\_Suryawansh](https://discuss.elastic.co/u/Aishvarya_Suryawansh)\ **Replies:** 2\ **Last updated:** [September 2, 2020, 12:59pm UTC](https://discuss.elastic.co/t/can-we-send-logs-to-webhook-url/247100 "2020-09-02T12:59:06Z")
Can we have webhook URL as destination URL in outputs need to send logstash data to Victorops eg: something like this output { webhook { hosts =\> \[""\] } } [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=306) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=308)