# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=736 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 737 --- ## [Logstash matching on variable fields](https://discuss.elastic.co/t/logstash-matching-on-variable-fields/136887)
**Author:** [@kylegordon](https://discuss.elastic.co/u/kylegordon)\ **Replies:** 2\ **Last updated:** [June 22, 2018, 12:40pm UTC](https://discuss.elastic.co/t/logstash-matching-on-variable-fields/136887 "2018-06-22T12:40:47Z")
I have Logstash receiving variable form data over syslog, and I'm having some issues getting Grok to match it. Once I've grabbed the relevant bits of syslog data, I'm left with the following types of messages. \[message… --- ## [Help required for grok filter creation \[windows platform\]](https://discuss.elastic.co/t/help-required-for-grok-filter-creation-windows-platform/137010)
**Author:** [@siddharthsk](https://discuss.elastic.co/u/siddharthsk)\ **Replies:** 1\ **Last updated:** [June 22, 2018, 12:26pm UTC](https://discuss.elastic.co/t/help-required-for-grok-filter-creation-windows-platform/137010 "2018-06-22T12:26:25Z")
Hi! I am new to elastic search and using Elastic search 6.2.3. I want create grok pattern for my following log format: \[Tue May 22 2018 15:16:56:325\] \[10.144.244.48\] \[CM\] \[12836\] \[Debug\] \[Starting Connection Manager e… --- ## [Filter data already exists into ES](https://discuss.elastic.co/t/filter-data-already-exists-into-es/136994)
**Author:** [@kimfut](https://discuss.elastic.co/u/kimfut)\ **Replies:** 0\ **Last updated:** [June 22, 2018, 9:00am UTC](https://discuss.elastic.co/t/filter-data-already-exists-into-es/136994 "2018-06-22T09:00:12Z")
hello every one ; i would like to know if is it possible to apply grok filter for data already exists into elasticsearch and filtered before , for your information i have ELK stack via docker , so everytime i do docker… --- ## [Logstash Configuration error in ubuntu?](https://discuss.elastic.co/t/logstash-configuration-error-in-ubuntu/136980)
**Author:** [@kartheek91](https://discuss.elastic.co/u/kartheek91)\ **Replies:** 1\ **Last updated:** [June 22, 2018, 8:48am UTC](https://discuss.elastic.co/t/logstash-configuration-error-in-ubuntu/136980 "2018-06-22T08:48:24Z")
Hi All actually I working on logstash to migrate sql server data to elasticsearch.So it worked in windows and now I'm trying to migrate logstash to ubuntu machine with same configuration file.But it is throwing me this e… --- ## [Http\_host only binds to 127.0.0.1](https://discuss.elastic.co/t/http-host-only-binds-to-127-0-0-1/136835)
**Author:** [@juanccert](https://discuss.elastic.co/u/juanccert)\ **Replies:** 4\ **Last updated:** [June 22, 2018, 8:09am UTC](https://discuss.elastic.co/t/http-host-only-binds-to-127-0-0-1/136835 "2018-06-22T08:09:18Z")
I'm trying to communicate my logstash instance with another host in a different LAN but I can't get my instance work in a different IP than the localhost: 127.0.0.1. This is my config settings: http.host:"172.20.25.… --- ## [Get the most recent event among event where have the same ID](https://discuss.elastic.co/t/get-the-most-recent-event-among-event-where-have-the-same-id/136657)
**Author:** [@Pierre2](https://discuss.elastic.co/u/Pierre2)\ **Replies:** 5\ **Last updated:** [June 22, 2018, 6:45am UTC](https://discuss.elastic.co/t/get-the-most-recent-event-among-event-where-have-the-same-id/136657 "2018-06-22T06:45:15Z")
Hello, My problem is that my logs line are composed with a @timestamp and an ID and I want to add a field to the line where is the most recent among lines with the same ID. There is an example of what I want. If there i… --- ## [Output file configuration](https://discuss.elastic.co/t/output-file-configuration/136269)
**Author:** [@antonopo](https://discuss.elastic.co/u/antonopo)\ **Replies:** 22\ **Last updated:** [June 22, 2018, 6:24am UTC](https://discuss.elastic.co/t/output-file-configuration/136269 "2018-06-22T06:24:06Z")
Hi, I have this configuration on 30-elasticsearch-output.conf output { elasticsearch { hosts =\> \["localhost:9200"\] sniffing =\> true manage\_template =\> false index =\> "%{\[@metadata\]\[beat\]}-%{+YYYY.MM.dd}" document… --- ## [Joining Multpile CSV files and injecting the data into Elasticsearch](https://discuss.elastic.co/t/joining-multpile-csv-files-and-injecting-the-data-into-elasticsearch/136957)
**Author:** [@Rakesh5216](https://discuss.elastic.co/u/Rakesh5216)\ **Replies:** 1\ **Last updated:** [June 22, 2018, 6:06am UTC](https://discuss.elastic.co/t/joining-multpile-csv-files-and-injecting-the-data-into-elasticsearch/136957 "2018-06-22T06:06:03Z")
Hello All, I am sure that you cant use "joins" in Kibana. I have a task which has to combine 3 csv files and inject the data into a single index on kibana. What are the different ways to solve this problem? I would app… --- ## [How to update a index based on fields other than the document\_id using logstash?](https://discuss.elastic.co/t/how-to-update-a-index-based-on-fields-other-than-the-document-id-using-logstash/133481)
**Author:** [@Rajkumar\_Selvakumar](https://discuss.elastic.co/u/Rajkumar_Selvakumar)\ **Replies:** 6\ **Last updated:** [June 22, 2018, 6:05am UTC](https://discuss.elastic.co/t/how-to-update-a-index-based-on-fields-other-than-the-document-id-using-logstash/133481 "2018-06-22T06:05:09Z")
Here is my use case, I have created a elastic index ( name : order\_index) and I able to load the index with the inputs from my database through logstash jdbc plugin ( all is fine ). the index looks like this seq\_id (d… --- ## [Logstash S3 output plugin fails if bucket name do not have a underscore (\_)](https://discuss.elastic.co/t/logstash-s3-output-plugin-fails-if-bucket-name-do-not-have-a-underscore/136826)
**Author:** [@thnitendra](https://discuss.elastic.co/u/thnitendra)\ **Replies:** 2\ **Last updated:** [June 22, 2018, 5:58am UTC](https://discuss.elastic.co/t/logstash-s3-output-plugin-fails-if-bucket-name-do-not-have-a-underscore/136826 "2018-06-22T05:58:35Z")
I am using logstash s3 output plugin to upoad files on ganesha s3 buckets. I must include an underscore in the bucket name to make the logstash S3 output plugin work. If there is no underscore, I get this error: :erro… --- ## [Is it possible to use split filter instead of grok in Logstash to chop message?](https://discuss.elastic.co/t/is-it-possible-to-use-split-filter-instead-of-grok-in-logstash-to-chop-message/136878)
**Author:** [@elasticheart](https://discuss.elastic.co/u/elasticheart)\ **Replies:** 5\ **Last updated:** [June 22, 2018, 4:36am UTC](https://discuss.elastic.co/t/is-it-possible-to-use-split-filter-instead-of-grok-in-logstash-to-chop-message/136878 "2018-06-22T04:36:57Z")
Hi, I am using ELK GA 6.3. I am using Logstash to read data from Kafka. My Kafka message is like; \ \ \ \ I am using the below grok pattern to split data; grok { matc… --- ## [How to optimize the multiple configuration files to improve the performance for too many filters](https://discuss.elastic.co/t/how-to-optimize-the-multiple-configuration-files-to-improve-the-performance-for-too-many-filters/136723)
**Author:** [@hyzhang6639](https://discuss.elastic.co/u/hyzhang6639)\ **Replies:** 8\ **Last updated:** [June 22, 2018, 3:38am UTC](https://discuss.elastic.co/t/how-to-optimize-the-multiple-configuration-files-to-improve-the-performance-for-too-many-filters/136723 "2018-06-22T03:38:14Z")
We have several configuration files include about 50 filters. It take about 9 mins to filter 10,000,000 events. After reducing to 30 filters, it take around 6 mins to complete to filter 10,000,000 events. In the future… --- ## [Logtsash decode incorrectly avro messages](https://discuss.elastic.co/t/logtsash-decode-incorrectly-avro-messages/136949)
**Author:** [@gergaith](https://discuss.elastic.co/u/gergaith)\ **Replies:** 0\ **Last updated:** [June 21, 2018, 10:42pm UTC](https://discuss.elastic.co/t/logtsash-decode-incorrectly-avro-messages/136949 "2018-06-21T22:42:30Z")
i'm trying to index data from kafka to ElasticSearch through logstash versions : logstash : 6.2.4 elasticSearch : 6.2.4 kafka : 0.10 avro : 1.8.2 the problem is that logstash read correctly the avro schema ( at lea… --- ## [Installing from Package Repository installs with missing core files](https://discuss.elastic.co/t/installing-from-package-repository-installs-with-missing-core-files/136947)
**Author:** [@alilland](https://discuss.elastic.co/u/alilland)\ **Replies:** 0\ **Last updated:** [June 21, 2018, 10:06pm UTC](https://discuss.elastic.co/t/installing-from-package-repository-installs-with-missing-core-files/136947 "2018-06-21T22:06:24Z")
After several attempts and several uninstalls, this problem keeps on persisting with installation -- it appears that the installation is missing some core files from the installation source $ sudo apt-get update && sudo… --- ## [How to extract things from beginning and end of a long single line log not parsing or extracting the thing in the middle of log](https://discuss.elastic.co/t/how-to-extract-things-from-beginning-and-end-of-a-long-single-line-log-not-parsing-or-extracting-the-thing-in-the-middle-of-log/136770)
**Author:** [@lserranov](https://discuss.elastic.co/u/lserranov)\ **Replies:** 4\ **Last updated:** [June 21, 2018, 9:49pm UTC](https://discuss.elastic.co/t/how-to-extract-things-from-beginning-and-end-of-a-long-single-line-log-not-parsing-or-extracting-the-thing-in-the-middle-of-log/136770 "2018-06-21T21:49:21Z")
Hi, Suppose that I have a single line log. This log has like 500 words. Suppose that I'm interested in extracting the initial 5 words and the last 5 words and I'm not interested on parsing the middle. Is there a way … --- ## [Logstash is failing](https://discuss.elastic.co/t/logstash-is-failing/136933)
**Author:** [@sandeepnarla22322](https://discuss.elastic.co/u/sandeepnarla22322)\ **Replies:** 7\ **Last updated:** [June 21, 2018, 8:17pm UTC](https://discuss.elastic.co/t/logstash-is-failing/136933 "2018-06-21T20:17:53Z")
/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:385\] elasticsearch - retrying failed action with response code: 403 ({"type"=\>"cluster\_block\_exception", "reason"=\>"blocked by: \[FORBIDDEN/12/index read-only / a… --- ## [Logstash not parsing new files added to docker volume](https://discuss.elastic.co/t/logstash-not-parsing-new-files-added-to-docker-volume/136796)
**Author:** [@Vikrant\_Aggarwal](https://discuss.elastic.co/u/Vikrant_Aggarwal)\ **Replies:** 4\ **Last updated:** [June 21, 2018, 7:04pm UTC](https://discuss.elastic.co/t/logstash-not-parsing-new-files-added-to-docker-volume/136796 "2018-06-21T19:04:50Z")
I have created a docker-compose file for ELK stack to parse the logs. \[root@onw-kwah-2v ELK-compose\]# cat docker-compose.yml version: '3' services: elasticsearch: image: docker.elastic.co/elasticsearch/elasticsear… --- ## [Exclude a file based on a value inside this file](https://discuss.elastic.co/t/exclude-a-file-based-on-a-value-inside-this-file/136307)
**Author:** [@dyl](https://discuss.elastic.co/u/dyl)\ **Replies:** 8\ **Last updated:** [June 21, 2018, 6:43pm UTC](https://discuss.elastic.co/t/exclude-a-file-based-on-a-value-inside-this-file/136307 "2018-06-21T18:43:35Z")
Hey, I listen several files in a directory thanks to Filebeat and then I parse those who interest me. For exemple : This file doesn't interest me because its "CODE RETOUR" value is 0 (last line) : INFO;0000;000002;\* … --- ## [Using ruby to clean part of a message](https://discuss.elastic.co/t/using-ruby-to-clean-part-of-a-message/136845)
**Author:** [@bbicen](https://discuss.elastic.co/u/bbicen)\ **Replies:** 3\ **Last updated:** [June 21, 2018, 6:09pm UTC](https://discuss.elastic.co/t/using-ruby-to-clean-part-of-a-message/136845 "2018-06-21T18:09:13Z")
Hi All, We are trying to send messages from an application server to Kafka, but some rows are too long so that we have to get rid of some part of the row. We can filter out long messages by ruby but we need to process f… --- ## [TIMESTAMP\_ISO8601 in date not working](https://discuss.elastic.co/t/timestamp-iso8601-in-date-not-working/136923)
**Author:** [@punit\_j](https://discuss.elastic.co/u/punit_j)\ **Replies:** 1\ **Last updated:** [June 21, 2018, 5:57pm UTC](https://discuss.elastic.co/t/timestamp-iso8601-in-date-not-working/136923 "2018-06-21T17:57:37Z")
I have a date as below: 2018-06-21 13:25:09-04:00 my logstash filter for date is as below: date { match =\> \[ "DeviceDate", "TIMESTAMP\_ISO8601"\] target =\> "@timestamp" } however i get below error on : Cannot creat… --- ## [Logstash cluster and single execution of cron trigger](https://discuss.elastic.co/t/logstash-cluster-and-single-execution-of-cron-trigger/136776)
**Author:** [@dbdbdb](https://discuss.elastic.co/u/dbdbdb)\ **Replies:** 2\ **Last updated:** [June 21, 2018, 5:18pm UTC](https://discuss.elastic.co/t/logstash-cluster-and-single-execution-of-cron-trigger/136776 "2018-06-21T17:18:19Z")
Hi, Is it possible to configure a Logstash cluster and execute a cron trigger only once instead in all logstash nodes? thanks! --- ## [Output-jdbc plugin crash on 6.3.0](https://discuss.elastic.co/t/output-jdbc-plugin-crash-on-6-3-0/136912)
**Author:** [@UVizor](https://discuss.elastic.co/u/UVizor)\ **Replies:** 1\ **Last updated:** [June 21, 2018, 4:38pm UTC](https://discuss.elastic.co/t/output-jdbc-plugin-crash-on-6-3-0/136912 "2018-06-21T16:38:48Z")
Hi! I updated my ELK from version 6.2.3 to 6.3.0 and in 6.3.0 logstash-output-jdbc can't starts. In logsthash-plain. log i have next errors: \[2018-06-21T17:16:00,948\]\[ERROR\]\[logstash.agent \] Failed to execute… --- ## [Logstash will not start with netflow module](https://discuss.elastic.co/t/logstash-will-not-start-with-netflow-module/136151)
**Author:** [@jkevan](https://discuss.elastic.co/u/jkevan)\ **Replies:** 3\ **Last updated:** [June 21, 2018, 3:59pm UTC](https://discuss.elastic.co/t/logstash-will-not-start-with-netflow-module/136151 "2018-06-21T15:59:28Z")
I have Logstash installed as a service. It starts as root so i can bind privileged ports. i successfully ran the bin/logstash --modules netflow --setup -M "netflow.var.kibana.host=10.x.x.x:5601" -M "netflow.var.input.u… --- ## [Error while uploading a Logfile using Logstash](https://discuss.elastic.co/t/error-while-uploading-a-logfile-using-logstash/135692)
**Author:** [@helloWorld1](https://discuss.elastic.co/u/helloWorld1)\ **Replies:** 10\ **Last updated:** [June 21, 2018, 2:27pm UTC](https://discuss.elastic.co/t/error-while-uploading-a-logfile-using-logstash/135692 "2018-06-21T14:27:41Z")
Hi All, I have simple logs in the format: \<\[package.filename\]\> The problem I faced is that the Logfile sometimes fails to get uploaded and i have to make changes in the file to get it uploaded. Moreover capital let… --- ## [Kv alphabetical sorting](https://discuss.elastic.co/t/kv-alphabetical-sorting/136876)
**Author:** [@jonviously](https://discuss.elastic.co/u/jonviously)\ **Replies:** 2\ **Last updated:** [June 21, 2018, 2:27pm UTC](https://discuss.elastic.co/t/kv-alphabetical-sorting/136876 "2018-06-21T14:27:01Z")
Hi ! I just discovered that the kv filter plugin applies an alphabetical sorting on keys in output. For example, i got a string like "a=1&c=2&b=3" after the kv plugin i got this : { "a" : "1" "b" : "3" "c" : "2" … --- ## [Extract fields from a log path in logstash](https://discuss.elastic.co/t/extract-fields-from-a-log-path-in-logstash/136754)
**Author:** [@aysala](https://discuss.elastic.co/u/aysala)\ **Replies:** 2\ **Last updated:** [June 21, 2018, 1:56pm UTC](https://discuss.elastic.co/t/extract-fields-from-a-log-path-in-logstash/136754 "2018-06-21T13:56:44Z")
Hi experts, I am trying to figure out a way to create new fields from a log path. Is there a way to configure this using processors in filebeat? or can anyone help me with the grok pattern that creates the below fields? … --- ## [Date extraction from a log file](https://discuss.elastic.co/t/date-extraction-from-a-log-file/136883)
**Author:** [@mariyana](https://discuss.elastic.co/u/mariyana)\ **Replies:** 1\ **Last updated:** [June 21, 2018, 1:43pm UTC](https://discuss.elastic.co/t/date-extraction-from-a-log-file/136883 "2018-06-21T13:43:05Z")
hi, I want to extract date from a log file to another field in elastic search that is coming through logstash. My log file is like this 21 May 2018 12:13:32,025 INFO \[org.jboss.as.configadmin\] (ServerService Thread Po… --- ## [Grok regex to filter custom data](https://discuss.elastic.co/t/grok-regex-to-filter-custom-data/136842)
**Author:** [@radumalica](https://discuss.elastic.co/u/radumalica)\ **Replies:** 0\ **Last updated:** [June 21, 2018, 11:55am UTC](https://discuss.elastic.co/t/grok-regex-to-filter-custom-data/136842 "2018-06-21T11:55:18Z")
Hi I am trying to define a grok to parse the following line: (It's from an Alcatel Lucent 7210SAS device) Jun 21 12:10:49 10.20.30.40 TMNX: 7655 Base SYSTEM-WARNING-ssiSyncConfigOK-2012 \[Syncronization of Config files\]… --- ## [Grock filter to parse SQL query](https://discuss.elastic.co/t/grock-filter-to-parse-sql-query/136721)
**Author:** [@wafaAG](https://discuss.elastic.co/u/wafaAG)\ **Replies:** 4\ **Last updated:** [June 21, 2018, 11:26am UTC](https://discuss.elastic.co/t/grock-filter-to-parse-sql-query/136721 "2018-06-21T11:26:32Z")
Hi, i have some events on application logs that i want to parse (which logging slow SQL query), the events is like that : 19-Jun-2018 15:16:28.670 WARNING \[ajp-nio-8009-exec-67\] org.apache.tomcat.jdbc.pool.interceptor.… --- ## [Logstash field value](https://discuss.elastic.co/t/logstash-field-value/136836)
**Author:** [@Raj\_Kumar](https://discuss.elastic.co/u/Raj_Kumar)\ **Replies:** 0\ **Last updated:** [June 21, 2018, 11:17am UTC](https://discuss.elastic.co/t/logstash-field-value/136836 "2018-06-21T11:17:32Z")
Hi There, I have a document which has a field values like this dns\_zone\_message1 : "1.3.19.14 (none)." dns\_zone\_message2 : 318186d0e64c021bfb95140287ebe32754" (none). dns\_zone\_message3: (none).regionh.top.local. (non… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=735) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=737)