# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=664 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 665 --- ## [How to remove Last Character from message](https://discuss.elastic.co/t/how-to-remove-last-character-from-message/153285)
**Author:** [@syedmujtaba88](https://discuss.elastic.co/u/syedmujtaba88)\ **Replies:** 1\ **Last updated:** [October 21, 2018, 4:52pm UTC](https://discuss.elastic.co/t/how-to-remove-last-character-from-message/153285 "2018-10-21T16:52:25Z")
i have a log like below for which i can successfully apply a grok pattern and decode the data to field. #### \ \<\> \<\> \< SOAP- \\> however my requirement is to capture the complete \ … --- ## [Logstash output flow stops to ES when using pipeline-to-pipeline isolator pattern and one output fails](https://discuss.elastic.co/t/logstash-output-flow-stops-to-es-when-using-pipeline-to-pipeline-isolator-pattern-and-one-output-fails/153289)
**Author:** [@wasserman](https://discuss.elastic.co/u/wasserman)\ **Replies:** 0\ **Last updated:** [October 21, 2018, 3:29pm UTC](https://discuss.elastic.co/t/logstash-output-flow-stops-to-es-when-using-pipeline-to-pipeline-isolator-pattern-and-one-output-fails/153289 "2018-10-21T15:29:19Z")
I followed the Logstash Output Isolator Pattern to push data to multiple Elastic stacks. Logstash 6.3.2 is pushing to ES 6.3.2 and ES 6.2.3. The ES 6.2.3 cluster is production and we want to add HA while migrating to E… --- ## [Logstash 6.4.2 Failed to execute action](https://discuss.elastic.co/t/logstash-6-4-2-failed-to-execute-action/153085)
**Author:** [@jpsilliman](https://discuss.elastic.co/u/jpsilliman)\ **Replies:** 1\ **Last updated:** [October 21, 2018, 3:30pm UTC](https://discuss.elastic.co/t/logstash-6-4-2-failed-to-execute-action/153085 "2018-10-21T15:30:56Z")
I am unable to resolve a config file error. Please advise if you have any suggestions. Error: \[2018-10-18T22:58:49,787\]\[ERROR\]\[logstash.agent \] Failed to execute action {:action=\>LogStash::PipelineAction::Cr… --- ## [Grok space expression optional](https://discuss.elastic.co/t/grok-space-expression-optional/152631)
**Author:** [@rotem\_lom](https://discuss.elastic.co/u/rotem_lom)\ **Replies:** 2\ **Last updated:** [October 21, 2018, 12:34pm UTC](https://discuss.elastic.co/t/grok-space-expression-optional/152631 "2018-10-21T12:34:06Z")
Hi there, In my environments I have the following log files: Domain name: WORKGROUP Console user name:Administrator Console user groups: \[Administrators,\] Logged in users: \[Administrator,\] OS version: Windows 10 OS Pl… --- ## [Install logstatsh plugin](https://discuss.elastic.co/t/install-logstatsh-plugin/151359)
**Author:** [@herzel](https://discuss.elastic.co/u/herzel)\ **Replies:** 4\ **Last updated:** [October 21, 2018, 11:41am UTC](https://discuss.elastic.co/t/install-logstatsh-plugin/151359 "2018-10-21T11:41:58Z")
Hi There. thanks in advance for your help. trying to install logstatsh plugin (any one) saying to me: \[elasticsearch@elklnx01 logstash-6.3.0\]$ bin/logstash-plugin install logstash-input-jms Validating logstash-input… --- ## [Logstash don't send data to elasticsearch](https://discuss.elastic.co/t/logstash-dont-send-data-to-elasticsearch/153200)
**Author:** [@safaa](https://discuss.elastic.co/u/safaa)\ **Replies:** 3\ **Last updated:** [October 20, 2018, 6:28pm UTC](https://discuss.elastic.co/t/logstash-dont-send-data-to-elasticsearch/153200 "2018-10-20T18:28:09Z")
I parsed a simple xml file . in first time the logstash parsed the file and sent data to elasticsearch . After that i deleted the index and i tried to parse it again , but logstash do not send data since i have deleted … --- ## [Codec Json not sending data to elasticsearch](https://discuss.elastic.co/t/codec-json-not-sending-data-to-elasticsearch/153018)
**Author:** [@dshetty](https://discuss.elastic.co/u/dshetty)\ **Replies:** 1\ **Last updated:** [October 20, 2018, 12:00am UTC](https://discuss.elastic.co/t/codec-json-not-sending-data-to-elasticsearch/153018 "2018-10-20T00:00:19Z")
INPUT: json {"userid": 125,"type": "SELL"} {"userid": 127,"type": "SELL"} LOGSTASH CONF FILE: input { kafka { bootstrap\_servers =\> "" topics =\> \["topic1"\] codec =\> "json" } } output { amazon\_es { hosts… --- ## [Filebeat to logstash geo\_point missing](https://discuss.elastic.co/t/filebeat-to-logstash-geo-point-missing/153080)
**Author:** [@rajsolanki](https://discuss.elastic.co/u/rajsolanki)\ **Replies:** 5\ **Last updated:** [October 19, 2018, 2:21pm UTC](https://discuss.elastic.co/t/filebeat-to-logstash-geo-point-missing/153080 "2018-10-19T14:21:36Z")
Ok Please pardon me for asking same question regarding geo\_point. I have done tons of reading in this forum and have tried everything i can think of after reading but i just cant find a solution to my issue. So here it … --- ## [Get the max integer of an array](https://discuss.elastic.co/t/get-the-max-integer-of-an-array/153178)
**Author:** [@Pierre2](https://discuss.elastic.co/u/Pierre2)\ **Replies:** 0\ **Last updated:** [October 19, 2018, 1:31pm UTC](https://discuss.elastic.co/t/get-the-max-integer-of-an-array/153178 "2018-10-19T13:31:27Z")
Hello, I'm looking for a way to get the max element of an array containing integer. I can't find any way to do that. Here is how I created my array in a ruby filter : DateTab=\[event.get('dateTransfertConseiller').to\_… --- ## [Include part(s) of configuration in conf file](https://discuss.elastic.co/t/include-part-s-of-configuration-in-conf-file/151851)
**Author:** [@parosio](https://discuss.elastic.co/u/parosio)\ **Replies:** 1\ **Last updated:** [October 19, 2018, 1:20pm UTC](https://discuss.elastic.co/t/include-part-s-of-configuration-in-conf-file/151851 "2018-10-19T13:20:07Z")
Hello, I'm aware it's possible to have logstash configured with a merge of different files found in a directory, but I've got a rather different structure for input files (and, consequently, for configurations). Nevert… --- ## [Unstructured DNS logs to grok or dissect](https://discuss.elastic.co/t/unstructured-dns-logs-to-grok-or-dissect/153149)
**Author:** [@bternyak](https://discuss.elastic.co/u/bternyak)\ **Replies:** 2\ **Last updated:** [October 19, 2018, 1:10pm UTC](https://discuss.elastic.co/t/unstructured-dns-logs-to-grok-or-dissect/153149 "2018-10-19T13:10:33Z")
Hy guys! I would approciate it if somebody could help me in this problem! I have DNS server logs, here is how it look: Oct 19 10:34:39 10.9.0.1 named\[24799\]: 19-Oct-2018 10:34:39.456 client 193.224.130.162#35702 (50.2… --- ## [Issues with SQS input plugin](https://discuss.elastic.co/t/issues-with-sqs-input-plugin/153174)
**Author:** [@whume](https://discuss.elastic.co/u/whume)\ **Replies:** 0\ **Last updated:** [October 19, 2018, 12:26pm UTC](https://discuss.elastic.co/t/issues-with-sqs-input-plugin/153174 "2018-10-19T12:26:00Z")
I am attempting to poll for messages with the sqs input plugin. Polling works but if the payload contains multiple JSON payloads only the first is returned. Im scratching my head trying to troubleshoot this. Does anyone … --- ## [How to start parsing AWS load balancer logs from specified date](https://discuss.elastic.co/t/how-to-start-parsing-aws-load-balancer-logs-from-specified-date/152711)
**Author:** [@nitish](https://discuss.elastic.co/u/nitish)\ **Replies:** 1\ **Last updated:** [October 19, 2018, 10:53am UTC](https://discuss.elastic.co/t/how-to-start-parsing-aws-load-balancer-logs-from-specified-date/152711 "2018-10-19T10:53:51Z")
AWS LB logs are stored in S3 bucket at AWSLogs/xxxxxxxxx/elasticloadbalancing/region/2018/10/16/\. I want to skip old logs and start parsing from start of this month. Setting prefix AWSLogs/xxxxxxxxx/elasticlo… --- ## [Sql\_last\_value as new field](https://discuss.elastic.co/t/sql-last-value-as-new-field/153143)
**Author:** [@mwe](https://discuss.elastic.co/u/mwe)\ **Replies:** 2\ **Last updated:** [October 19, 2018, 9:11am UTC](https://discuss.elastic.co/t/sql-last-value-as-new-field/153143 "2018-10-19T09:11:41Z")
Hi, is it posible to use the value of sql\_last\_value in a new field ? i use the following config: input { jdbc { jdbc\_driver\_library =\> "/usr/local/lib/db2jcc4.jar,/usr/local/lib/db2jcc\_license\_cisuz.jar" jdbc\_d… --- ## [Grok pattern is matching but not working](https://discuss.elastic.co/t/grok-pattern-is-matching-but-not-working/153130)
**Author:** [@Diluka](https://discuss.elastic.co/u/Diluka)\ **Replies:** 9\ **Last updated:** [October 19, 2018, 8:42am UTC](https://discuss.elastic.co/t/grok-pattern-is-matching-but-not-working/153130 "2018-10-19T08:42:54Z")
config in pipeline/logstash.conf: filter { grok { match =\> { "message" =\> \[ "\\\[%{TIMESTAMP\_ISO8601:timestamp}\\\] \\\[%{LOGLEVEL:level}\\\] \\\[%{PATH:path}:%{POSINT:line}\\\] \\\[%{WORD:node}\\\] \\\[%{GREEDYDATA:message}\\\]"… --- ## [Drop filter for multiple names](https://discuss.elastic.co/t/drop-filter-for-multiple-names/153108)
**Author:** [@pavuk](https://discuss.elastic.co/u/pavuk)\ **Replies:** 2\ **Last updated:** [October 19, 2018, 8:32am UTC](https://discuss.elastic.co/t/drop-filter-for-multiple-names/153108 "2018-10-19T08:32:00Z")
Hi, Could you please help me with drop logstash filter? I have field 'username' and I want to drop all records for specific usernames. Please correct my filter if it is wrong: if \["Pete", "John", "Bill", "Greg", "Pau… --- ## [Logstash grok filters](https://discuss.elastic.co/t/logstash-grok-filters/153122)
**Author:** [@Gaurav\_Agarwal](https://discuss.elastic.co/u/Gaurav_Agarwal)\ **Replies:** 2\ **Last updated:** [October 19, 2018, 8:08am UTC](https://discuss.elastic.co/t/logstash-grok-filters/153122 "2018-10-19T08:08:22Z")
Hello I wanted to suppose we have a log and w define custom fields for the same and implement grok on it then on basis of a particular field as defined in the custom pattern how can we implement conditional cases on bas… --- ## [How to create if conditional statement in logstash config.yml file](https://discuss.elastic.co/t/how-to-create-if-conditional-statement-in-logstash-config-yml-file/153134)
**Author:** [@Nethaji](https://discuss.elastic.co/u/Nethaji)\ **Replies:** 1\ **Last updated:** [October 19, 2018, 8:04am UTC](https://discuss.elastic.co/t/how-to-create-if-conditional-statement-in-logstash-config-yml-file/153134 "2018-10-19T08:04:20Z")
Eg: if id =1 means i need to change this into "compliance" can u please explain how to do this? --- ## [Drop old messages using Ruby filter](https://discuss.elastic.co/t/drop-old-messages-using-ruby-filter/151990)
**Author:** [@programagor](https://discuss.elastic.co/u/programagor)\ **Replies:** 1\ **Last updated:** [October 19, 2018, 7:51am UTC](https://discuss.elastic.co/t/drop-old-messages-using-ruby-filter/151990 "2018-10-19T07:51:47Z")
Greetings Occasionally, my logstash receives a message from few days back, and logstash tries to write it into an index which was already marked as readonly and forcemerged. Logstash then receives a 403 error from Elast… --- ## [Multiple patterns in grok filter](https://discuss.elastic.co/t/multiple-patterns-in-grok-filter/153106)
**Author:** [@Billz1026](https://discuss.elastic.co/u/Billz1026)\ **Replies:** 3\ **Last updated:** [October 19, 2018, 6:05am UTC](https://discuss.elastic.co/t/multiple-patterns-in-grok-filter/153106 "2018-10-19T06:05:44Z")
Hi, I have written following grok filter to match multiple log formats. filter { grok { match =\> { "message" =\> "%{MONTH:Month}%{SPACE}%{NUMBER:Day} %{TIME:Time} %{IP:Host} %{NUMBER:SEQ\_NO}: %{DATA:Month\_From\_Device… --- ## [Using Logstash as relay](https://discuss.elastic.co/t/using-logstash-as-relay/153076)
**Author:** [@pas](https://discuss.elastic.co/u/pas)\ **Replies:** 1\ **Last updated:** [October 19, 2018, 5:44am UTC](https://discuss.elastic.co/t/using-logstash-as-relay/153076 "2018-10-19T05:44:41Z")
In perspective of security I like the principle of not exposing elastic-notes directly to data sources or end users. I therefore see it as relevant to use Logstash as a relay for data which should not be filtered/manipul… --- ## [Make Logstash drop documents on 403](https://discuss.elastic.co/t/make-logstash-drop-documents-on-403/149977)
**Author:** [@programagor](https://discuss.elastic.co/u/programagor)\ **Replies:** 8\ **Last updated:** [October 19, 2018, 5:24am UTC](https://discuss.elastic.co/t/make-logstash-drop-documents-on-403/149977 "2018-10-19T05:24:55Z")
Greetings Recently I started using forcemerge on my old indices. However, I found out that occasionally, Logstash writes into the older indices, increasing the segment count, so the curator has to merge them again on th… --- ## [Handle exceptions](https://discuss.elastic.co/t/handle-exceptions/150478)
**Author:** [@Sripal](https://discuss.elastic.co/u/Sripal)\ **Replies:** 7\ **Last updated:** [October 19, 2018, 5:15am UTC](https://discuss.elastic.co/t/handle-exceptions/150478 "2018-10-19T05:15:54Z")
Hi all, I have sysout log with many exception. how to read that exceptions. Is there any better way to read the exception. I'm using gork filter to read the pattern(am successfully able to read the pattern). But I can't… --- ## [Require suggestion for the logstash processing : Logstash](https://discuss.elastic.co/t/require-suggestion-for-the-logstash-processing-logstash/153097)
**Author:** [@triptapuri](https://discuss.elastic.co/u/triptapuri)\ **Replies:** 0\ **Last updated:** [October 19, 2018, 4:05am UTC](https://discuss.elastic.co/t/require-suggestion-for-the-logstash-processing-logstash/153097 "2018-10-19T04:05:43Z")
This error occured when i use logstash to collect logs and output it to Elastic Search. i use logstash on my linux server and logstash version is 6.4.2.Input file size is 40 gb and data is unstructure following my logs… --- ## [Cant parse rails logs](https://discuss.elastic.co/t/cant-parse-rails-logs/153095)
**Author:** [@JonArthur](https://discuss.elastic.co/u/JonArthur)\ **Replies:** 0\ **Last updated:** [October 19, 2018, 3:26am UTC](https://discuss.elastic.co/t/cant-parse-rails-logs/153095 "2018-10-19T03:26:11Z")
I, \[2018-05-03 02:02:06#1433\] INFO -- : {"method":"GET","path":"/api/v2/product\_categories/main","format":"json","controller":"Api::V2::ProductCategoriesController","action":"main","status":401,"duration":4.07,"view":0.0… --- ## [Cant‘s create logstash service in Centos 6](https://discuss.elastic.co/t/cant-s-create-logstash-service-in-centos-6/152144)
**Author:** [@Haitianisgood](https://discuss.elastic.co/u/Haitianisgood)\ **Replies:** 2\ **Last updated:** [October 19, 2018, 2:48am UTC](https://discuss.elastic.co/t/cant-s-create-logstash-service-in-centos-6/152144 "2018-10-19T02:48:24Z")
ENV: CentOS release 6.8 (Final) 64bit logstash 6.4.2 I installed logstash with yum. I executed service logstash start,but it reported logstash: unrecognized service. I had executed initctl start logstash but not work… --- ## [Logstash 6.4.2 with modules netflow can't import date to es](https://discuss.elastic.co/t/logstash-6-4-2-with-modules-netflow-cant-import-date-to-es/152955)
**Author:** [@zqc0512](https://discuss.elastic.co/u/zqc0512)\ **Replies:** 1\ **Last updated:** [October 19, 2018, 12:30am UTC](https://discuss.elastic.co/t/logstash-6-4-2-with-modules-netflow-cant-import-date-to-es/152955 "2018-10-19T00:30:44Z")
logstash 6.4.2 with modules netflow can't import date to es. my logstash.yml file is modules: - name: netflow var.input.udp.port: 9996 var.elasticsearch.hosts: "11.11.xxx:9900,11.11.xx:9900,11.11.xx:9900,11.… --- ## [JDBC input plugin only works outside of pipeline - 6.4.2](https://discuss.elastic.co/t/jdbc-input-plugin-only-works-outside-of-pipeline-6-4-2/153074)
**Author:** [@RenataBoodram](https://discuss.elastic.co/u/RenataBoodram)\ **Replies:** 0\ **Last updated:** [October 18, 2018, 9:25pm UTC](https://discuss.elastic.co/t/jdbc-input-plugin-only-works-outside-of-pipeline-6-4-2/153074 "2018-10-18T21:25:03Z")
Please note: I am able to run this configuration successfully in Logstash 6.4.2 using the command logstash -f simple.conf but not using pipelines. indent preformatted text by 4 spaces Version: 6.4.2 Operating Syst… --- ## [Logstash pid file](https://discuss.elastic.co/t/logstash-pid-file/152891)
**Author:** [@shivaprasadrao.b](https://discuss.elastic.co/u/shivaprasadrao.b)\ **Replies:** 3\ **Last updated:** [October 18, 2018, 8:58pm UTC](https://discuss.elastic.co/t/logstash-pid-file/152891 "2018-10-18T20:58:53Z")
Hi All, i'm starting my logstash with command ./logstash -f logstash.conf & as a backend process. i would like to generate a pid file in my logstash home directory. How to achieve this. For kibana i have used pid.fil… --- ## [Multiline plugin for logstash not working for gelf input](https://discuss.elastic.co/t/multiline-plugin-for-logstash-not-working-for-gelf-input/152908)
**Author:** [@fewknow](https://discuss.elastic.co/u/fewknow)\ **Replies:** 11\ **Last updated:** [October 18, 2018, 8:45pm UTC](https://discuss.elastic.co/t/multiline-plugin-for-logstash-not-working-for-gelf-input/152908 "2018-10-18T20:45:54Z")
I have configured my logstash inputs as: input { stdin { } gelf { host =\> "0.0.0.0" port =\> 12201 codec =\> multiline { pattern =\> "^\\s" what =\> "previous" } } udp { codec =\> json… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=663) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=665)