# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=448 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 449 --- ## [Grok add field after CSV Filter](https://discuss.elastic.co/t/grok-add-field-after-csv-filter/207366)
**Author:** [@khaag](https://discuss.elastic.co/u/khaag)\ **Replies:** 0\ **Last updated:** [November 11, 2019, 2:14pm UTC](https://discuss.elastic.co/t/grok-add-field-after-csv-filter/207366 "2019-11-11T14:14:31Z")
Hi I have a working csv filter for a syslog event from Palo Alto (SYSTEM) else if (\[message\] =~ /globalprotect/) { csv { source =\> "message" columns =\> \[ "FUTURE\_USE", "Receive Time", "Serial Number", "Type", "Cont… --- ## [Mutate object field](https://discuss.elastic.co/t/mutate-object-field/207356)
**Author:** [@KrisKarrots](https://discuss.elastic.co/u/KrisKarrots)\ **Replies:** 0\ **Last updated:** [November 11, 2019, 1:33pm UTC](https://discuss.elastic.co/t/mutate-object-field/207356 "2019-11-11T13:33:14Z")
Hello, I am trying to mutate an object field (JSON object) using the add\_field to rename it, after this has been applied, the new field looks like this {message=1},{message=2},{message=3},{message=4} - in one line Pre… --- ## [Changing field Name](https://discuss.elastic.co/t/changing-field-name/207124)
**Author:** [@Prashant\_Kumar3](https://discuss.elastic.co/u/Prashant_Kumar3)\ **Replies:** 1\ **Last updated:** [November 11, 2019, 1:32pm UTC](https://discuss.elastic.co/t/changing-field-name/207124 "2019-11-11T13:32:27Z")
Hi everybody, I am getting a field name and value like this: How\_to\_configure\_outlook\_SOP\_name:How to configure outlook How\_to\_enable\_macros\_SOP\_name: How to enable macros. e.g. (anything)\_SOP\_name:value I put all t… --- ## [How to create a custom document](https://discuss.elastic.co/t/how-to-create-a-custom-document/206820)
**Author:** [@Kernel\_Panic](https://discuss.elastic.co/u/Kernel_Panic)\ **Replies:** 4\ **Last updated:** [November 11, 2019, 12:26pm UTC](https://discuss.elastic.co/t/how-to-create-a-custom-document/206820 "2019-11-11T12:26:09Z")
Hello I was able to configure the following: From the json document I have this: "MyValue": "Key value based on a specific field", "host": { "name": "hostname" }, "tags": \[ "beats\_input\_codec\_plain\_applied", \], … --- ## [Logstash fails to write in configured Index](https://discuss.elastic.co/t/logstash-fails-to-write-in-configured-index/207332)
**Author:** [@Moritz\_Kiesewetter](https://discuss.elastic.co/u/Moritz_Kiesewetter)\ **Replies:** 0\ **Last updated:** [November 11, 2019, 10:45am UTC](https://discuss.elastic.co/t/logstash-fails-to-write-in-configured-index/207332 "2019-11-11T10:45:21Z")
Hello guys, i have the problem, that i want to manage my Index Names by Date. output { if \[type\] == "syslog" { elasticsearch { hosts =\> \["XX.XX.XXX.XXX:ZZZZ", "XX.XX.XXX.XXX:ZZZZ", "XX.XX.XXX.XXX:ZZZZ"\] index… --- ## [Logstash - Arcsight integration](https://discuss.elastic.co/t/logstash-arcsight-integration/207323)
**Author:** [@daniele.saccon](https://discuss.elastic.co/u/daniele.saccon)\ **Replies:** 0\ **Last updated:** [November 11, 2019, 10:08am UTC](https://discuss.elastic.co/t/logstash-arcsight-integration/207323 "2019-11-11T10:08:31Z")
Hi everybody I activated the Arcsight module and created a pipeline for the configurations already active in logstash (in my case I receive data from the Beats). When I activate Arcsight the pipeline with the Beats does… --- ## [Create index dynamically](https://discuss.elastic.co/t/create-index-dynamically/207315)
**Author:** [@Anuwa\_A](https://discuss.elastic.co/u/Anuwa_A)\ **Replies:** 0\ **Last updated:** [November 11, 2019, 9:31am UTC](https://discuss.elastic.co/t/create-index-dynamically/207315 "2019-11-11T09:31:36Z")
I want to create ES index in logstash conf file dynamically based on id, if first character of id = 1 i need to create my index as 1\_myindex and if id = 2 create my index as 2\_myindex. Thanks --- ## [Parasing Custom Logs](https://discuss.elastic.co/t/parasing-custom-logs/207270)
**Author:** [@kadiveti](https://discuss.elastic.co/u/kadiveti)\ **Replies:** 1\ **Last updated:** [November 11, 2019, 8:55am UTC](https://discuss.elastic.co/t/parasing-custom-logs/207270 "2019-11-11T08:55:28Z")
Hi Team We are trying to parase custom logs. But the pattern of the logs changes with Newline or space sometime a new Is getting. Is there a way to get the logs Irrespective of data changes. Thanks --- ## [Nmap result xml file send to elasticsearch using logstash - read\_to\_eof Error](https://discuss.elastic.co/t/nmap-result-xml-file-send-to-elasticsearch-using-logstash-read-to-eof-error/207308)
**Author:** [@GwakMin](https://discuss.elastic.co/u/GwakMin)\ **Replies:** 0\ **Last updated:** [November 11, 2019, 8:51am UTC](https://discuss.elastic.co/t/nmap-result-xml-file-send-to-elasticsearch-using-logstash-read-to-eof-error/207308 "2019-11-11T08:51:28Z")
I want to send to nmap xml result to elasticsearch using logstash. So i wrote nmap-logstash conf file like below input { file { path =\> "/home/elktest/elk/nmaptoelk/\*.xml" start\_position =\> "beginning" si… --- ## [Logs are not parsing properly](https://discuss.elastic.co/t/logs-are-not-parsing-properly/207307)
**Author:** [@Sundaramoorthy\_Anand](https://discuss.elastic.co/u/Sundaramoorthy_Anand)\ **Replies:** 0\ **Last updated:** [November 11, 2019, 8:48am UTC](https://discuss.elastic.co/t/logs-are-not-parsing-properly/207307 "2019-11-11T08:48:28Z")
My architecture is \*Filebeat A\*(remote) \> Logstash A (2 pipelines) \> Elasticsearch A \> Kibana A \*Filebeat B\*(remote) \> Logstash A (2 pipelines) \> Elasticsearch A \> Kibana A Its for logs analysis. say my logs format i… --- ## [Systemctl logstash.service pipeline problem](https://discuss.elastic.co/t/systemctl-logstash-service-pipeline-problem/206892)
**Author:** [@basavyr](https://discuss.elastic.co/u/basavyr)\ **Replies:** 7\ **Last updated:** [November 11, 2019, 7:29am UTC](https://discuss.elastic.co/t/systemctl-logstash-service-pipeline-problem/206892 "2019-11-11T07:29:54Z")
Ok, so I've been dealing with this issue for a couple of days and it's becoming really frustrating. I want to start logstash with systemctl start logstash.service to have it in the background sending some metric logs to… --- ## [How to integrate logstash in nodejs application?](https://discuss.elastic.co/t/how-to-integrate-logstash-in-nodejs-application/207238)
**Author:** [@Shubham\_Akodiya](https://discuss.elastic.co/u/Shubham_Akodiya)\ **Replies:** 1\ **Last updated:** [November 11, 2019, 5:38am UTC](https://discuss.elastic.co/t/how-to-integrate-logstash-in-nodejs-application/207238 "2019-11-11T05:38:47Z")
I'm working on node application and my main goal is to maintain the logs (error, info) of the backend part in logstash so that I could do some analysis of which api is breaking and why. I'm new to logstash and I read som… --- ## [Logstash Template Format For Adding Multiple Fields of the Same Type](https://discuss.elastic.co/t/logstash-template-format-for-adding-multiple-fields-of-the-same-type/207263)
**Author:** [@scott\_stash](https://discuss.elastic.co/u/scott_stash)\ **Replies:** 0\ **Last updated:** [November 11, 2019, 2:28am UTC](https://discuss.elastic.co/t/logstash-template-format-for-adding-multiple-fields-of-the-same-type/207263 "2019-11-11T02:28:39Z")
My template file is becoming very large as I have had to add a lot of fields to it of Long and Text, so I have the following in the file over and over again: (for text and long types) "server": { "ty… --- ## [Replacing part of one field's string with value from another field](https://discuss.elastic.co/t/replacing-part-of-one-fields-string-with-value-from-another-field/207257)
**Author:** [@saif3r](https://discuss.elastic.co/u/saif3r)\ **Replies:** 0\ **Last updated:** [November 10, 2019, 10:19pm UTC](https://discuss.elastic.co/t/replacing-part-of-one-fields-string-with-value-from-another-field/207257 "2019-11-10T22:19:29Z")
Hi, I'm parsing an XML and I currently have the following array: errorinfo.Msg { "Index": \[ "1" \], "Display": \[ "0" \], "Name": \[ "Movement too slow" \], "Text": \[ "1 Movement too slow con… --- ## [Logstash parsing mysql-slow.log](https://discuss.elastic.co/t/logstash-parsing-mysql-slow-log/206597)
**Author:** [@Vladpov](https://discuss.elastic.co/u/Vladpov)\ **Replies:** 10\ **Last updated:** [November 10, 2019, 5:50pm UTC](https://discuss.elastic.co/t/logstash-parsing-mysql-slow-log/206597 "2019-11-10T17:50:59Z")
Hey guys, I'm a little bit struggling with Logtash parsing mysql-slow.log file. I configured Filebeat to send logs from the file into Elasticsearch through Logstash so I can see recieved messages in Kibana like this: N… --- ## [How to rename dynamical fields](https://discuss.elastic.co/t/how-to-rename-dynamical-fields/207245)
**Author:** [@ja123](https://discuss.elastic.co/u/ja123)\ **Replies:** 1\ **Last updated:** [November 10, 2019, 4:03pm UTC](https://discuss.elastic.co/t/how-to-rename-dynamical-fields/207245 "2019-11-10T16:03:29Z")
I have got the following filterd xml and I would like to rename the field fpDate for every single element, but I don't know how to interate through the \[parsed\]\[Journey\]. The size of the \[Journey\] array is always 20 elem… --- ## [Parse rancher logs with logstash](https://discuss.elastic.co/t/parse-rancher-logs-with-logstash/207242)
**Author:** [@tru64gurus](https://discuss.elastic.co/u/tru64gurus)\ **Replies:** 0\ **Last updated:** [November 10, 2019, 2:04pm UTC](https://discuss.elastic.co/t/parse-rancher-logs-with-logstash/207242 "2019-11-10T14:04:56Z")
Hi, I have serveral k8s clusters running on rancher 2.3.1 sending several GB of logs per second and causing disk pressure on source side . To solve source bottlneck , logs are being sent to syslog server and get writte… --- ## [How to parse field name with '.'](https://discuss.elastic.co/t/how-to-parse-field-name-with/207241)
**Author:** [@111233](https://discuss.elastic.co/u/111233)\ **Replies:** 0\ **Last updated:** [November 10, 2019, 1:41pm UTC](https://discuss.elastic.co/t/how-to-parse-field-name-with/207241 "2019-11-10T13:41:22Z")
"error"=\>{"type"=\>"mapper\_parsing\_exception", "reason"=\>"Field name \[Sea Scallops N.Y. Times Recipe\] cannot contain '.'"} Are there any solutions to this problem?? --- ## [Create index by aggregating existing index](https://discuss.elastic.co/t/create-index-by-aggregating-existing-index/207011)
**Author:** [@aasoft](https://discuss.elastic.co/u/aasoft)\ **Replies:** 6\ **Last updated:** [November 9, 2019, 10:22pm UTC](https://discuss.elastic.co/t/create-index-by-aggregating-existing-index/207011 "2019-11-09T22:22:55Z")
I created an index as below: "hits" : \[ { "\_index" : "family", "\_type" : "doc", "\_id" : "1", "\_score" : 1.0, "\_source" : { "fkm\_family" : "1", "pkm\_article" : "1001" } }, { "\_index" : "family", "\_type" : "d… --- ## [Kafka Logstash Plugin unavailable broker (swaps broker addresses for discovery?)](https://discuss.elastic.co/t/kafka-logstash-plugin-unavailable-broker-swaps-broker-addresses-for-discovery/207220)
**Author:** [@jbpratt](https://discuss.elastic.co/u/jbpratt)\ **Replies:** 5\ **Last updated:** [November 9, 2019, 9:23pm UTC](https://discuss.elastic.co/t/kafka-logstash-plugin-unavailable-broker-swaps-broker-addresses-for-discovery/207220 "2019-11-09T21:23:05Z")
Hey everyone, I'm pretty new to the stack and ran into some issues with trying to use a Kafka input. Here is my logstash.conf and logs from attaching to the container. In the logs, it successfully subscribes to the topi… --- ## [Parsing xml with 3 different variants of its content](https://discuss.elastic.co/t/parsing-xml-with-3-different-variants-of-its-content/207179)
**Author:** [@saif3r](https://discuss.elastic.co/u/saif3r)\ **Replies:** 4\ **Last updated:** [November 9, 2019, 8:11pm UTC](https://discuss.elastic.co/t/parsing-xml-with-3-different-variants-of-its-content/207179 "2019-11-09T20:11:38Z")
Hi, One of our production machines outputs the log into an xml file. Said xml can have 3 different structures: \ ... \ ... \ ... \ or \ ... \ or \ ..… --- ## [Logstash not loading csv file to elasticsearch](https://discuss.elastic.co/t/logstash-not-loading-csv-file-to-elasticsearch/206651)
**Author:** [@Gowtham\_Gandham](https://discuss.elastic.co/u/Gowtham_Gandham)\ **Replies:** 2\ **Last updated:** [November 9, 2019, 5:17pm UTC](https://discuss.elastic.co/t/logstash-not-loading-csv-file-to-elasticsearch/206651 "2019-11-09T17:17:25Z")
input { file { path =\> "/home/sigma/Desktop/docker-elk/logstash/\*.csv" start\_position =\> "beginning" sincedb\_path =\> "/dev/null" } } ## Add your filters / logstash plugins configuration here filter { csv { separ… --- ## [Struggling to parse JSON data with logstash](https://discuss.elastic.co/t/struggling-to-parse-json-data-with-logstash/207186)
**Author:** [@Lee\_Quince](https://discuss.elastic.co/u/Lee_Quince)\ **Replies:** 4\ **Last updated:** [November 9, 2019, 3:21pm UTC](https://discuss.elastic.co/t/struggling-to-parse-json-data-with-logstash/207186 "2019-11-09T15:21:39Z")
So I have a basic http json post that is sent on the format of { "sq": 3, "ht": \[ { "si": "90:B6:86:42:2C:9F", "sm": "90B686", "bi": "00:00:00:00:00:00", "ap": 0, "ot": 1473246736, "ct": 1473246736, "sh": -90, … --- ## [Logstash json to multiple documents](https://discuss.elastic.co/t/logstash-json-to-multiple-documents/207007)
**Author:** [@vikasp](https://discuss.elastic.co/u/vikasp)\ **Replies:** 8\ **Last updated:** [November 9, 2019, 2:18pm UTC](https://discuss.elastic.co/t/logstash-json-to-multiple-documents/207007 "2019-11-09T14:18:49Z")
I have an input in the below format: \\\1\\this is first event\\\\2\\this is second event\\\\3\\this is third event\>\\\\4\\<… --- ## [Parse JSON](https://discuss.elastic.co/t/parse-json/207104)
**Author:** [@Buzz](https://discuss.elastic.co/u/Buzz)\ **Replies:** 9\ **Last updated:** [November 9, 2019, 2:13pm UTC](https://discuss.elastic.co/t/parse-json/207104 "2019-11-09T14:13:58Z")
Hi All, I would like to ask you for a help with parsing data from JSON. I have a log file with huge JSON that contains non parsed parts: dataj.AffectedItems { "Attachments": "ATT00001.png (24733b); ATT00002.jpg (24684… --- ## [Batch size in logstash](https://discuss.elastic.co/t/batch-size-in-logstash/206919)
**Author:** [@Ankit\_Sawant](https://discuss.elastic.co/u/Ankit_Sawant)\ **Replies:** 7\ **Last updated:** [November 9, 2019, 7:57am UTC](https://discuss.elastic.co/t/batch-size-in-logstash/206919 "2019-11-09T07:57:34Z")
I have 4 events having unique Id and I am processing these event in aggregate filter but with batch size of 250. Logstash is processing these events out of order I am not getting exact result, with batch size of 2 the ev… --- ## [Jdbc connection](https://discuss.elastic.co/t/jdbc-connection/207118)
**Author:** [@mbrown007](https://discuss.elastic.co/u/mbrown007)\ **Replies:** 6\ **Last updated:** [November 9, 2019, 3:30am UTC](https://discuss.elastic.co/t/jdbc-connection/207118 "2019-11-09T03:30:44Z")
How do we know which jar version to use with a particular version of java and logstash? --- ## [How to tell if tags is empty or does not exists?](https://discuss.elastic.co/t/how-to-tell-if-tags-is-empty-or-does-not-exists/207168)
**Author:** [@mtudisco](https://discuss.elastic.co/u/mtudisco)\ **Replies:** 2\ **Last updated:** [November 8, 2019, 9:18pm UTC](https://discuss.elastic.co/t/how-to-tell-if-tags-is-empty-or-does-not-exists/207168 "2019-11-08T21:18:32Z")
Hi, I want to be able to check if tags field is empty or if it does not exists in order to send the logstash output to different indexes. I'm using logstash 7.3 output { stdout { codec =\> rubydebug } if \[tags\] == \[… --- ## [Logstash to logstash communication](https://discuss.elastic.co/t/logstash-to-logstash-communication/207164)
**Author:** [@Senthil\_ak](https://discuss.elastic.co/u/Senthil_ak)\ **Replies:** 0\ **Last updated:** [November 8, 2019, 7:51pm UTC](https://discuss.elastic.co/t/logstash-to-logstash-communication/207164 "2019-11-08T19:51:39Z")
Hi, I am sending logs from one logstash to another through TCP or UDP. Earlier I tried with TCP i got error which posted at https://discuss.elastic.co/t/eoferror-end-of-file-reached/204933 so switched to UDP but now I a… --- ## [Mutate rename filter not working on nested fields](https://discuss.elastic.co/t/mutate-rename-filter-not-working-on-nested-fields/207146)
**Author:** [@Justin\_Doles](https://discuss.elastic.co/u/Justin_Doles)\ **Replies:** 2\ **Last updated:** [November 8, 2019, 7:16pm UTC](https://discuss.elastic.co/t/mutate-rename-filter-not-working-on-nested-fields/207146 "2019-11-08T19:16:28Z")
I'm likely missing something simple, but I cannot seem to get the rename filter to work with my data. The field I'm attempting to rename is nested. It doesn't error, but it also doesn't do anything. I am able to renam… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=447) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=449)