# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1017 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1018 --- ## [Logstash Update or Insert - logic for updating if existing](https://discuss.elastic.co/t/logstash-update-or-insert-logic-for-updating-if-existing/78189)
**Author:** [@whyapenny](https://discuss.elastic.co/u/whyapenny)\ **Replies:** 3\ **Last updated:** [March 16, 2017, 3:49pm UTC](https://discuss.elastic.co/t/logstash-update-or-insert-logic-for-updating-if-existing/78189 "2017-03-16T15:49:24Z")
Hey guys, I am having a bit of an issue attempting to apply some logic to mimic a previous mysql functionality. The idea is that after parsing i create a unique id, Doc\_id of 3 fields concatenated. Then i query … --- ## [Ruby code not updating integer](https://discuss.elastic.co/t/ruby-code-not-updating-integer/78791)
**Author:** [@whyapenny](https://discuss.elastic.co/u/whyapenny)\ **Replies:** 2\ **Last updated:** [March 16, 2017, 3:29pm UTC](https://discuss.elastic.co/t/ruby-code-not-updating-integer/78791 "2017-03-16T15:29:47Z")
i am attempting to take a field and increment the number by one. I had assumed this would be an easy task but it appears to be not so trivial. I have resorted to using the ruby filter and attempting to do this here: … --- ## [Always create fields defined by Xpath](https://discuss.elastic.co/t/always-create-fields-defined-by-xpath/78875)
**Author:** [@jcannell](https://discuss.elastic.co/u/jcannell)\ **Replies:** 0\ **Last updated:** [March 16, 2017, 2:10pm UTC](https://discuss.elastic.co/t/always-create-fields-defined-by-xpath/78875 "2017-03-16T14:10:11Z")
Hello, I'm using the xml filter plugin in logstash. Specifically, I've been using the xpath setting to parse my xml data into fields. I wanted to know if it's possible for logstash to create the field I've defined in… --- ## [\_geoip\_lookup\_failure](https://discuss.elastic.co/t/-geoip-lookup-failure/78873)
**Author:** [@m.francia](https://discuss.elastic.co/u/m.francia)\ **Replies:** 0\ **Last updated:** [March 16, 2017, 1:59pm UTC](https://discuss.elastic.co/t/-geoip-lookup-failure/78873 "2017-03-16T13:59:06Z")
Hi! I have this grok pattern: RTMPLOGV %{IPORHOST:ipclient} \\\[%{HTTPDATE:timestamp}\\\] %{WORD:command} "%{GREEDYDATA:app}" "%{WORD:name}" %{DATA} %{NUMBER:bytesreceived} %{NUMBER:bytessent} "%{URI:url}" "%{HOSTNAME:fl… --- ## [Problem with GeoIP using filebeat-\>logstash-\>ES-\>Kibana (No geo\_point type)](https://discuss.elastic.co/t/problem-with-geoip-using-filebeat-logstash-es-kibana-no-geo-point-type/78763)
**Author:** [@zokratez](https://discuss.elastic.co/u/zokratez)\ **Replies:** 3\ **Last updated:** [March 16, 2017, 1:36pm UTC](https://discuss.elastic.co/t/problem-with-geoip-using-filebeat-logstash-es-kibana-no-geo-point-type/78763 "2017-03-16T13:36:09Z")
This is my config: filter { if \[type\] == "apache-access" { grok { match =\> \[ "message", "%{IP:clientip} %{USER:ident} %{USER:auth} \\\[%{HTTPDATE:apache\_timestamp}\\\] \\"%{WORD:method} /%{NOTSPACE:request\_page}… --- ## [Logstash Failed to create monitoring event](https://discuss.elastic.co/t/logstash-failed-to-create-monitoring-event/74191)
**Author:** [@faustf](https://discuss.elastic.co/u/faustf)\ **Replies:** 5\ **Last updated:** [March 16, 2017, 1:12pm UTC](https://discuss.elastic.co/t/logstash-failed-to-create-monitoring-event/74191 "2017-03-16T13:12:15Z")
Hy guys After I installed x-pack for monitoring logstash - elasticsearch and kibana, I've the following error on logstash: \[2017-02-07T10:18:05,181\]\[ERROR\]\[logstash.inputs.metrics \] Failed to create monitoring event {… --- ## [Read CSV without introducing, and having to remove, a message field?](https://discuss.elastic.co/t/read-csv-without-introducing-and-having-to-remove-a-message-field/78809)
**Author:** [@GrahamHannington](https://discuss.elastic.co/u/GrahamHannington)\ **Replies:** 1\ **Last updated:** [March 16, 2017, 1:03pm UTC](https://discuss.elastic.co/t/read-csv-without-introducing-and-having-to-remove-a-message-field/78809 "2017-03-16T13:03:21Z")
I can use Logstash to read a file of JSON Lines from stdin without introducing—and, hence, without needing to remove—a message field: input { stdin { codec =\> json\_lines } } (No filter section required; the con… --- ## [Add field from csv to event log send by logstash](https://discuss.elastic.co/t/add-field-from-csv-to-event-log-send-by-logstash/77677)
**Author:** [@Grenouille06](https://discuss.elastic.co/u/Grenouille06)\ **Replies:** 23\ **Last updated:** [March 16, 2017, 10:01am UTC](https://discuss.elastic.co/t/add-field-from-csv-to-event-log-send-by-logstash/77677 "2017-03-16T10:01:34Z")
Hi friends, I have a question about filters. I have a conf file that works perfectly for sending active directory logs to elasticsearch. In my log, I have a field named "event\_data.TargetUserName and it has a registrat… --- ## [Trying to Grok a field from String to Int](https://discuss.elastic.co/t/trying-to-grok-a-field-from-string-to-int/78736)
**Author:** [@tomer](https://discuss.elastic.co/u/tomer)\ **Replies:** 5\ **Last updated:** [March 16, 2017, 8:23am UTC](https://discuss.elastic.co/t/trying-to-grok-a-field-from-string-to-int/78736 "2017-03-16T08:23:17Z")
Hi, I have a log as such: { "recordType":"MT", "callingNumber":"5555555", "callingImsi":"", "callingMsc":"", "billable":"", "calledNumber":"5555555", "calledImsi":"425240787504111", "calledMsc":"972723800111", … --- ## [Persistent Queues - invitation for beta testing!](https://discuss.elastic.co/t/persistent-queues-invitation-for-beta-testing/72705)
**Author:** [@alvin](https://discuss.elastic.co/u/alvin)\ **Replies:** 5\ **Last updated:** [March 16, 2017, 7:39am UTC](https://discuss.elastic.co/t/persistent-queues-invitation-for-beta-testing/72705 "2017-03-16T07:39:56Z")
Logstash 5.1 introduced an exciting new feature called Persistent Queues (PQs), which provides two main benefits to Logstash users: Message durability - protection against in-flight data loss. All incoming events wil… --- ## [How to parse array of arrays inside json](https://discuss.elastic.co/t/how-to-parse-array-of-arrays-inside-json/78658)
**Author:** [@vaishnavi](https://discuss.elastic.co/u/vaishnavi)\ **Replies:** 4\ **Last updated:** [March 16, 2017, 6:24am UTC](https://discuss.elastic.co/t/how-to-parse-array-of-arrays-inside-json/78658 "2017-03-16T06:24:55Z")
Hi I am trying to parse json which contains arrays within arrays and want it give them as seperate records in elasticsearch using logstash My input is \[{"root":\[{"data":\[{"id1" : \[{"value":"1"}\],"id2" : \[{"value":"… --- ## [Problem with Output section](https://discuss.elastic.co/t/problem-with-output-section/78786)
**Author:** [@GregMeadows](https://discuss.elastic.co/u/GregMeadows)\ **Replies:** 3\ **Last updated:** [March 16, 2017, 5:31am UTC](https://discuss.elastic.co/t/problem-with-output-section/78786 "2017-03-16T05:31:47Z")
I am building a conf file to use a template for all other csv ingestion to ES indexes I need. This conf was working (in a simpler) form and is based on a v2 dev logstash install. Question... Has the syntax changed when… --- ## [ES Logstash config problem](https://discuss.elastic.co/t/es-logstash-config-problem/78328)
**Author:** [@Tigryss](https://discuss.elastic.co/u/Tigryss)\ **Replies:** 3\ **Last updated:** [March 16, 2017, 4:50am UTC](https://discuss.elastic.co/t/es-logstash-config-problem/78328 "2017-03-16T04:50:13Z")
hi where can i set pool size , active threads , queued tasks i get this error \[2017-03-13T12:08:29,434\]\[INFO \]\[logstash.outputs.elasticsearch\] retrying failed action with response code: 429 ({"type"=\>"es\_rejected\_… --- ## [Aggregate and elapsed filter plugins - multiple events processing](https://discuss.elastic.co/t/aggregate-and-elapsed-filter-plugins-multiple-events-processing/78573)
**Author:** [@Nikolay\_Shushkin](https://discuss.elastic.co/u/Nikolay_Shushkin)\ **Replies:** 1\ **Last updated:** [March 16, 2017, 4:44am UTC](https://discuss.elastic.co/t/aggregate-and-elapsed-filter-plugins-multiple-events-processing/78573 "2017-03-16T04:44:56Z")
Why there is a limitation in aggregate plugin usage: "You should be very careful to set logstash filter workers to 1 (-w 1 flag) for this filter to work correctly otherwise events may be processed out of sequence and une… --- ## [Logs latency](https://discuss.elastic.co/t/logs-latency/78688)
**Author:** [@ruby](https://discuss.elastic.co/u/ruby)\ **Replies:** 1\ **Last updated:** [March 16, 2017, 4:38am UTC](https://discuss.elastic.co/t/logs-latency/78688 "2017-03-16T04:38:41Z")
No records of logs in logstash.log file, and on kibana logs are showing but very late.. --- ## [Translate multiple fields in an event](https://discuss.elastic.co/t/translate-multiple-fields-in-an-event/78502)
**Author:** [@tsangdl](https://discuss.elastic.co/u/tsangdl)\ **Replies:** 4\ **Last updated:** [March 16, 2017, 4:35am UTC](https://discuss.elastic.co/t/translate-multiple-fields-in-an-event/78502 "2017-03-16T04:35:29Z")
Hi, I have events going into logstash. Each event has a number of fields two of which I would like to translate The fields are customer\_id and integrator\_id. I would like to add fields customer\_name and integrator\_name… --- ## [Logstash with Filebeat transfer over TLS](https://discuss.elastic.co/t/logstash-with-filebeat-transfer-over-tls/78794)
**Author:** [@pleegor](https://discuss.elastic.co/u/pleegor)\ **Replies:** 0\ **Last updated:** [March 16, 2017, 4:08am UTC](https://discuss.elastic.co/t/logstash-with-filebeat-transfer-over-tls/78794 "2017-03-16T04:08:51Z")
Hi, I have configured ELK and client with filebeat as 2 VMs on Vagnrant/VirtualBox. When enable TLS I am not able ship logs to logstash. /etc/logstash/conf.d/02-beats-input.conf input { beats { port =\> 5044 … --- ## [Filebeat connection to ELK via port 5044](https://discuss.elastic.co/t/filebeat-connection-to-elk-via-port-5044/78781)
**Author:** [@pleegor](https://discuss.elastic.co/u/pleegor)\ **Replies:** 9\ **Last updated:** [March 16, 2017, 4:07am UTC](https://discuss.elastic.co/t/filebeat-connection-to-elk-via-port-5044/78781 "2017-03-16T04:07:46Z")
Hi, I am trying to setup a filebeat shipping logs to ELK instance. 2017/03/16 01:07:45.254081 output.go:109: DBG output worker: publish 2046 events 2017/03/16 01:07:45.254086 sync.go:53: DBG connect 2017… --- ## [Behavior of sqs input plugin when output plugin fails](https://discuss.elastic.co/t/behavior-of-sqs-input-plugin-when-output-plugin-fails/78793)
**Author:** [@mnozawa](https://discuss.elastic.co/u/mnozawa)\ **Replies:** 0\ **Last updated:** [March 16, 2017, 3:59am UTC](https://discuss.elastic.co/t/behavior-of-sqs-input-plugin-when-output-plugin-fails/78793 "2017-03-16T03:59:22Z")
Hi everyone I am using sqs input plugin and elasticsearch output plugin (with no filter). When indexing to elasticsearch fails because of no running target cluster, the document ("message" in sqs term) seems lost. My… --- ## [Logstash multiple output endpoints single event](https://discuss.elastic.co/t/logstash-multiple-output-endpoints-single-event/78751)
**Author:** [@Exocomp](https://discuss.elastic.co/u/Exocomp)\ **Replies:** 1\ **Last updated:** [March 15, 2017, 9:06pm UTC](https://discuss.elastic.co/t/logstash-multiple-output-endpoints-single-event/78751 "2017-03-15T21:06:57Z")
Hi, I've got a use case where for a single event I need to send data to elasticsearch as well a udp server. The udp server receives input only in a certain format so I don't want to send it everything I'm sending elas… --- ## [Using percolate in a Logstash filter?](https://discuss.elastic.co/t/using-percolate-in-a-logstash-filter/76381)
**Author:** [@es435](https://discuss.elastic.co/u/es435)\ **Replies:** 5\ **Last updated:** [March 16, 2017, 12:25am UTC](https://discuss.elastic.co/t/using-percolate-in-a-logstash-filter/76381 "2017-03-16T00:25:36Z")
I have an index of percolator queries on Elasticsearch and I'm trying to write a Logstash config file with a filter that will allow me to percolate documents against those queries as I index them, sending an alert if the… --- ## [Grok Parse failure with fail2ban](https://discuss.elastic.co/t/grok-parse-failure-with-fail2ban/77492)
**Author:** [@xon](https://discuss.elastic.co/u/xon)\ **Replies:** 7\ **Last updated:** [March 15, 2017, 9:30pm UTC](https://discuss.elastic.co/t/grok-parse-failure-with-fail2ban/77492 "2017-03-15T21:30:00Z")
I followed 2 different guides and I still havent figure out where the issue is. I want to parse fail2ban logs and I tried the following two configurations unsuccessfully! Pattern files that I tried root@leeds:/etc/log… --- ## [Logstash kafka input consumer group not sharing partitions](https://discuss.elastic.co/t/logstash-kafka-input-consumer-group-not-sharing-partitions/77705)
**Author:** [@Maxwell\_Flanders](https://discuss.elastic.co/u/Maxwell_Flanders)\ **Replies:** 2\ **Last updated:** [March 15, 2017, 8:09pm UTC](https://discuss.elastic.co/t/logstash-kafka-input-consumer-group-not-sharing-partitions/77705 "2017-03-15T20:09:13Z")
I have a really weird case where I have a 6 logstash consumers consuming from a kafka topic using the logstash-kafka-input plugin. The 6 logstash consumers are their own logstash instances on different machines (and have… --- ## [Cipher filter: multiple source?](https://discuss.elastic.co/t/cipher-filter-multiple-source/78728)
**Author:** [@Vittorio](https://discuss.elastic.co/u/Vittorio)\ **Replies:** 1\ **Last updated:** [March 15, 2017, 7:45pm UTC](https://discuss.elastic.co/t/cipher-filter-multiple-source/78728 "2017-03-15T19:45:20Z")
Hi all, Is it possible to specify multiple source using cipher filter? I guess the answer is no , because the source field is a string.... But Is that possible anybody never had this problem? thanks --- ## [Adding a field from existing ones](https://discuss.elastic.co/t/adding-a-field-from-existing-ones/77697)
**Author:** [@Mohamed](https://discuss.elastic.co/u/Mohamed)\ **Replies:** 8\ **Last updated:** [March 15, 2017, 6:45pm UTC](https://discuss.elastic.co/t/adding-a-field-from-existing-ones/77697 "2017-03-15T18:45:57Z")
Hello, I want to add a field composed of some other fields, like this: suppose we have fields "name": \[ "XX" \], "adress": \[ "584" \], I want to add a field like that: "p… --- ## [Logstash csv - \[\[main\]\ **Author:** [@alexus](https://discuss.elastic.co/u/alexus)\ **Replies:** 2\ **Last updated:** [March 9, 2017, 3:25pm UTC](https://discuss.elastic.co/t/logstash-csv-main-file-debug-logstash-inputs-file-each-file-grew/77898 "2017-03-09T15:25:37Z") I'm trying to input csv file into elasticsearch, however before going there, I'd like to display data on screen first, yet when I attempt to run logstash, I'm getting an error and cannot figure out what I'm doing wrong..… --- ## [Wrong monitoring statistics - jvm heap size](https://discuss.elastic.co/t/wrong-monitoring-statistics-jvm-heap-size/78655)
**Author:** [@jetnet](https://discuss.elastic.co/u/jetnet)\ **Replies:** 12\ **Last updated:** [March 15, 2017, 4:50pm UTC](https://discuss.elastic.co/t/wrong-monitoring-statistics-jvm-heap-size/78655 "2017-03-15T16:50:05Z")
Hi All, I just compared the jvm heap usage from kibana and visualvm and realized, that Kibana shows incorrect values. LS: v.5.2.2 LS\_HEAP\_SIZE=512m Charts: \[image\] … --- ## [Conditionals with multiple strings](https://discuss.elastic.co/t/conditionals-with-multiple-strings/78726)
**Author:** [@jcannell](https://discuss.elastic.co/u/jcannell)\ **Replies:** 2\ **Last updated:** [March 15, 2017, 4:40pm UTC](https://discuss.elastic.co/t/conditionals-with-multiple-strings/78726 "2017-03-15T16:40:50Z")
Hello, I'm trying to drop an event should there be either "Device Product" or "Device Vendor" in the \[message\] field. This works fine: if ("Device Vendor" in \[message\] or "Device Product" in \[message\]) { d… --- ## [File output with append only files](https://discuss.elastic.co/t/file-output-with-append-only-files/78064)
**Author:** [@erikanderson753](https://discuss.elastic.co/u/erikanderson753)\ **Replies:** 1\ **Last updated:** [March 15, 2017, 4:21pm UTC](https://discuss.elastic.co/t/file-output-with-append-only-files/78064 "2017-03-15T16:21:58Z")
We are experimenting with setting up a file output to store certain types of logs. Right after the file is created by logstash we are using incron to set the file to append only with chattr +a. Logstash writes to the … --- ## [Where can I find the default template?](https://discuss.elastic.co/t/where-can-i-find-the-default-template/78618)
**Author:** [@camkidman](https://discuss.elastic.co/u/camkidman)\ **Replies:** 5\ **Last updated:** [March 15, 2017, 3:51pm UTC](https://discuss.elastic.co/t/where-can-i-find-the-default-template/78618 "2017-03-15T15:51:30Z")
Hi there, I've got the following in my elasticsearch configuration file: output { elasticsearch { hosts =\> \["https://...."\] index =\> "production-%{+YYYY.MM.dd}" } stdout { codec =\> rubydebug } } I'm runni… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1016) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1018)