# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=906 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 907 --- ## [Grok filter output to elastic not filtered](https://discuss.elastic.co/t/grok-filter-output-to-elastic-not-filtered/100898)
**Author:** [@fridolf](https://discuss.elastic.co/u/fridolf)\ **Replies:** 2\ **Last updated:** [September 20, 2017, 8:02am UTC](https://discuss.elastic.co/t/grok-filter-output-to-elastic-not-filtered/100898 "2017-09-20T08:02:14Z")
Using logstash 5.4.0 and elastic 5.4.0 I have a grok filter: %{MYDATESTAMP:loggedtimestamp} GMT%{ISO8601\_TIMEZONE} %{MYTZ} \[-\] %{LOGLEVEL:loglevel}: %{GREEDYDATA:logmessage} I have below patterns: MYTZ (CE\[S\]?T) MYD… --- ## [Logstash filter conditions does not work,how to get The real user ipaddress from access log of Apache?](https://discuss.elastic.co/t/logstash-filter-conditions-does-not-work-how-to-get-the-real-user-ipaddress-from-access-log-of-apache/101091)
**Author:** [@jagerzhang](https://discuss.elastic.co/u/jagerzhang)\ **Replies:** 4\ **Last updated:** [September 20, 2017, 7:51am UTC](https://discuss.elastic.co/t/logstash-filter-conditions-does-not-work-how-to-get-the-real-user-ipaddress-from-access-log-of-apache/101091 "2017-09-20T07:51:52Z")
Logstash filter conditions does not work， as follows： filter { json { source =\> "message" } if "%{client\_ip}" == "-" { mutate { remove\_field =\> "client\_ip"… --- ## [KV field\_split prevents logstash ingesting data](https://discuss.elastic.co/t/kv-field-split-prevents-logstash-ingesting-data/100434)
**Author:** [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Replies:** 19\ **Last updated:** [September 20, 2017, 7:46am UTC](https://discuss.elastic.co/t/kv-field-split-prevents-logstash-ingesting-data/100434 "2017-09-20T07:46:06Z")
Hi, I found something strange going on with the field\_split option. I'm trying to parse fortigate logfiles. Default log: status=deny policyid=0 dst\_country="Reserved" src\_country="Reserved" service=1947/udp proto=17… --- ## [Ruby files in Logstash](https://discuss.elastic.co/t/ruby-files-in-logstash/101101)
**Author:** [@vinod\_hy](https://discuss.elastic.co/u/vinod_hy)\ **Replies:** 1\ **Last updated:** [September 20, 2017, 6:50am UTC](https://discuss.elastic.co/t/ruby-files-in-logstash/101101 "2017-09-20T06:50:42Z")
Hi All, I have logstash 5.4.0 version running on my machine. In that folder structure, i can see ruby files (.rb extension). Can you please tell me the significance of these files. Can i get a working logstash without … --- ## [Filters in logstash for ELK](https://discuss.elastic.co/t/filters-in-logstash-for-elk/98995)
**Author:** [@iam.Carrot](https://discuss.elastic.co/u/iam.Carrot)\ **Replies:** 30\ **Last updated:** [September 20, 2017, 4:56am UTC](https://discuss.elastic.co/t/filters-in-logstash-for-elk/98995 "2017-09-20T04:56:00Z")
Hi, I am relatively new to the ELK stack. I am writing up a logstash fconfig to push my logs to elastic index for kibana. Below is the case scenario: I am using the gork filter with match =\> { “message” =\> “%{COMBINEDA… --- ## [Logstash gsub remove string/characters](https://discuss.elastic.co/t/logstash-gsub-remove-string-characters/100852)
**Author:** [@tharu85](https://discuss.elastic.co/u/tharu85)\ **Replies:** 15\ **Last updated:** [September 20, 2017, 3:49am UTC](https://discuss.elastic.co/t/logstash-gsub-remove-string-characters/100852 "2017-09-20T03:49:57Z")
In my log file, I need to remove a defined characters from the RAW log. my RAW log sample as below. Sep 18 150942 db\[27414\]: category=\\"Event\\" subcategory=\\"System\\".... Sep 18 151043 db\[27464\]: category=\\"Event… --- ## [Windows logstash input section for adding new input type](https://discuss.elastic.co/t/windows-logstash-input-section-for-adding-new-input-type/101095)
**Author:** [@sam281](https://discuss.elastic.co/u/sam281)\ **Replies:** 0\ **Last updated:** [September 20, 2017, 3:48am UTC](https://discuss.elastic.co/t/windows-logstash-input-section-for-adding-new-input-type/101095 "2017-09-20T03:48:33Z")
Hello, I have a few remote windows machines forwarding OS events using the WEF feature to one central windows machine and I am trying to read that data through Logstash and forward it to my application. I see below erro… --- ## [Template Creation for type geo\_point in logstash](https://discuss.elastic.co/t/template-creation-for-type-geo-point-in-logstash/100967)
**Author:** [@shashwat](https://discuss.elastic.co/u/shashwat)\ **Replies:** 5\ **Last updated:** [September 20, 2017, 12:38am UTC](https://discuss.elastic.co/t/template-creation-for-type-geo-point-in-logstash/100967 "2017-09-20T00:38:27Z")
Hi, I have a csv file which contains Latitude\_(A),Longitude\_(A),Latitude\_(B),Longitude\_(B) fields. I want to use this data to put on coordinate map in Kibana. I tried with following configuration: mutate { convert =\> … --- ## [Is there an Application for Semi-Automatic Grok Construction?](https://discuss.elastic.co/t/is-there-an-application-for-semi-automatic-grok-construction/100908)
**Author:** [@Jeff\_Kubina](https://discuss.elastic.co/u/Jeff_Kubina)\ **Replies:** 5\ **Last updated:** [September 19, 2017, 9:11pm UTC](https://discuss.elastic.co/t/is-there-an-application-for-semi-automatic-grok-construction/100908 "2017-09-19T21:11:52Z")
As best I can tell there does not seem to be a tool or application that will in a semi-automatically, or in a semi-supervised manner, derive grok patterns from a large collection of log messages. I think such a tool woul… --- ## [Merge different log files to create single event](https://discuss.elastic.co/t/merge-different-log-files-to-create-single-event/101066)
**Author:** [@shashwat](https://discuss.elastic.co/u/shashwat)\ **Replies:** 0\ **Last updated:** [September 19, 2017, 7:32pm UTC](https://discuss.elastic.co/t/merge-different-log-files-to-create-single-event/101066 "2017-09-19T19:32:43Z")
Hi, I have three log files. Each being generated at different time in a day. Is there any option available in logstash where I can combine the records from each of the log file based on some common key present in each o… --- ## [\[Solved\] Index Patterns don't match](https://discuss.elastic.co/t/solved-index-patterns-dont-match/101059)
**Author:** [@KnechtionsCoding](https://discuss.elastic.co/u/KnechtionsCoding)\ **Replies:** 1\ **Last updated:** [September 19, 2017, 6:00pm UTC](https://discuss.elastic.co/t/solved-index-patterns-dont-match/101059 "2017-09-19T18:00:11Z")
Logs that are pushed from filebeat aren't showing up in logstash. My pipeline from logstash to elasticsearch works, as metric beat and heartbeat are getting pushed through, but my logs from filebeat are disappearing int… --- ## [Cant find installed plugin](https://discuss.elastic.co/t/cant-find-installed-plugin/97098)
**Author:** [@jaime\_loom](https://discuss.elastic.co/u/jaime_loom)\ **Replies:** 7\ **Last updated:** [September 19, 2017, 5:01pm UTC](https://discuss.elastic.co/t/cant-find-installed-plugin/97098 "2017-09-19T17:01:19Z")
Hi I am creating a new logstash output plugin and I’m trying to test it with logstash 5.5.1 I install it with bin/logstash-plugin install path/to/logstash-output-azure-0.1.1.gem the out put is Validating /path/to/logs… --- ## [Dead\_letter\_queue plugin issue](https://discuss.elastic.co/t/dead-letter-queue-plugin-issue/101054)
**Author:** [@godson2006](https://discuss.elastic.co/u/godson2006)\ **Replies:** 1\ **Last updated:** [September 19, 2017, 4:55pm UTC](https://discuss.elastic.co/t/dead-letter-queue-plugin-issue/101054 "2017-09-19T16:55:35Z")
Hello, I am currently having an issue with property sincedb\_path. My logstash service is dockerized and when using this plugin I do not have to implicitly set the sincedb\_path attribute because internally logstash will … --- ## [Logstash filter not applying to all records](https://discuss.elastic.co/t/logstash-filter-not-applying-to-all-records/100886)
**Author:** [@ananthasooner](https://discuss.elastic.co/u/ananthasooner)\ **Replies:** 4\ **Last updated:** [September 19, 2017, 3:44pm UTC](https://discuss.elastic.co/t/logstash-filter-not-applying-to-all-records/100886 "2017-09-19T15:44:20Z")
For each entry, i see in kibana, some have i\_hostname value and some don't, even though beat.hostname is available in each entry. Same happening with idtest, i am aware that idtest is based on condition, but still for th… --- ## [Grokparsefailure in logstash when consuming beats](https://discuss.elastic.co/t/grokparsefailure-in-logstash-when-consuming-beats/99891)
**Author:** [@bsmithlea](https://discuss.elastic.co/u/bsmithlea)\ **Replies:** 1\ **Last updated:** [September 11, 2017, 8:53pm UTC](https://discuss.elastic.co/t/grokparsefailure-in-logstash-when-consuming-beats/99891 "2017-09-11T20:53:53Z")
I am seeing \_grokparsefailure tags in my output from logstash even though all the fields are getting values. My input is from a filebeat source. Here is som json output from my filebeat. My logstash pipeline config look… --- ## [Geo\_Point field is not coming](https://discuss.elastic.co/t/geo-point-field-is-not-coming/100640)
**Author:** [@Shubham\_Mahajan](https://discuss.elastic.co/u/Shubham_Mahajan)\ **Replies:** 5\ **Last updated:** [September 19, 2017, 2:18pm UTC](https://discuss.elastic.co/t/geo-point-field-is-not-coming/100640 "2017-09-19T14:18:10Z")
Hi There, I am trying to parse apache logs using grok. but i am not seeing any geo-point type in my index. Please help. My template: { "template": "apache\_filebeat", "settings": { "number\_of\_shards": 1, … --- ## [Creating a Persistent / Custom Variable when parsing a log file](https://discuss.elastic.co/t/creating-a-persistent-custom-variable-when-parsing-a-log-file/101030)
**Author:** [@sgrover](https://discuss.elastic.co/u/sgrover)\ **Replies:** 0\ **Last updated:** [September 19, 2017, 1:40pm UTC](https://discuss.elastic.co/t/creating-a-persistent-custom-variable-when-parsing-a-log-file/101030 "2017-09-19T13:40:00Z")
Hi All - I have read through few questions and I think a similar question has been covered before - however I do not understand Ruby code to make sense of the responses. My problem is as follows: SOURCE FIELD I am pars… --- ## [Unable to update records using jdbc plugin](https://discuss.elastic.co/t/unable-to-update-records-using-jdbc-plugin/101024)
**Author:** [@kanchan](https://discuss.elastic.co/u/kanchan)\ **Replies:** 0\ **Last updated:** [September 19, 2017, 1:12pm UTC](https://discuss.elastic.co/t/unable-to-update-records-using-jdbc-plugin/101024 "2017-09-19T13:12:14Z")
Hi, I have create configuration file like: input { jdbc { jdbc\_validate\_connection =\> true jdbc\_connection\_string =\> "jdbc:oracle:thin:@MUMCHORA66.ad.crisil.com:1821/COA" jdbc\_user =\> "WALLET\_CARE" jdbc\_password =\>… --- ## [Twitter input raising errors when using "@" into the keyword config](https://discuss.elastic.co/t/twitter-input-raising-errors-when-using-into-the-keyword-config/99677)
**Author:** [@Melvyn\_Peignon](https://discuss.elastic.co/u/Melvyn_Peignon)\ **Replies:** 1\ **Last updated:** [September 19, 2017, 1:07pm UTC](https://discuss.elastic.co/t/twitter-input-raising-errors-when-using-into-the-keyword-config/99677 "2017-09-19T13:07:30Z")
I have a pipeline using twitter input: input { twitter { consumer\_key =\> "key" consumer\_secret =\> "key" oauth\_token =\> "key" oauth\_token\_secret =\> "key" keywords =\> \[ "Microsoft"\] f… --- ## [Parentchild relationship in logstash](https://discuss.elastic.co/t/parentchild-relationship-in-logstash/100207)
**Author:** [@swaplogstash](https://discuss.elastic.co/u/swaplogstash)\ **Replies:** 2\ **Last updated:** [September 19, 2017, 1:03pm UTC](https://discuss.elastic.co/t/parentchild-relationship-in-logstash/100207 "2017-09-19T13:03:10Z")
I am using Multiple Jdbc input plugins(since I have two Sqls , one is parent and other is child table) to load data into elastic which has parent child relationship between two mappings. Can you please prov… --- ## [Where to specify path.logs in Logstash 2.x?](https://discuss.elastic.co/t/where-to-specify-path-logs-in-logstash-2-x/100930)
**Author:** [@amruth](https://discuss.elastic.co/u/amruth)\ **Replies:** 6\ **Last updated:** [September 19, 2017, 10:45am UTC](https://discuss.elastic.co/t/where-to-specify-path-logs-in-logstash-2-x/100930 "2017-09-19T10:45:42Z")
Hi, I am using Logstash 2.4(since Java 1.7 doesn't support Logstash 5.x). Since it's Logstash 2.x there will be no Logstash.yml file. Now, I want to run Logstash as a service and wnat to gspecify options like path.logs,… --- ## [Understanding the aggregation filter:](https://discuss.elastic.co/t/understanding-the-aggregation-filter/100928)
**Author:** [@tomer](https://discuss.elastic.co/u/tomer)\ **Replies:** 0\ **Last updated:** [September 18, 2017, 8:08pm UTC](https://discuss.elastic.co/t/understanding-the-aggregation-filter/100928 "2017-09-18T20:08:45Z")
well I am able to aggregate that my last log has all the terms, but I can't know who will be my last log, therefore I did "push\_map\_as\_event\_on\_timeout =\> true" (I kept all config that is not repetitive since maybe some … --- ## [Logstash 5.0.2 can not install to specified path。。。](https://discuss.elastic.co/t/logstash-5-0-2-can-not-install-to-specified-path/100820)
**Author:** [@bejjm](https://discuss.elastic.co/u/bejjm)\ **Replies:** 1\ **Last updated:** [September 19, 2017, 10:14am UTC](https://discuss.elastic.co/t/logstash-5-0-2-can-not-install-to-specified-path/100820 "2017-09-19T10:14:19Z")
Hi all, I download logstash-5.0.2.rpm file . used command to install : sudo rpm -ivh --prefix=/opt logstash-5.0.2.rpm But，install logstash is failed . ERROR MESSAGE : \[root@cdm-autoinstall-single rpm\]# sudo rpm -iv… --- ## [Grok to match all instances](https://discuss.elastic.co/t/grok-to-match-all-instances/100887)
**Author:** [@elvarb](https://discuss.elastic.co/u/elvarb)\ **Replies:** 1\ **Last updated:** [September 19, 2017, 9:50am UTC](https://discuss.elastic.co/t/grok-to-match-all-instances/100887 "2017-09-19T09:50:27Z")
When I have a large error message with lots of error Id's I would want to be able to add all those ID's to a new field, a new field used similarly to the tag field. For example ORA-12012: error on auto execute of job O… --- ## [Make event wait](https://discuss.elastic.co/t/make-event-wait/100983)
**Author:** [@tomer](https://discuss.elastic.co/u/tomer)\ **Replies:** 5\ **Last updated:** [September 19, 2017, 8:30am UTC](https://discuss.elastic.co/t/make-event-wait/100983 "2017-09-19T08:30:11Z")
Hi, Can I make that if I get a log that parameter X = 'a' that Logstash will not push it straight to ES but wait for 3 minutes? --- ## [ConcurrencyError in Ruby filter plugin in Logstash](https://discuss.elastic.co/t/concurrencyerror-in-ruby-filter-plugin-in-logstash/100557)
**Author:** [@tkhan](https://discuss.elastic.co/u/tkhan)\ **Replies:** 0\ **Last updated:** [September 14, 2017, 2:27pm UTC](https://discuss.elastic.co/t/concurrencyerror-in-ruby-filter-plugin-in-logstash/100557 "2017-09-14T14:27:50Z")
We are using Elasticsearch, Logstash, and Filebeat to process log files of our application. Filebeat reads log messages from log files and sends those to Logstash. Logstash parses those logs and stores into Elasticsearch… --- ## [Grokparsefailure when field is no longer in the message](https://discuss.elastic.co/t/grokparsefailure-when-field-is-no-longer-in-the-message/100882)
**Author:** [@scayzer65](https://discuss.elastic.co/u/scayzer65)\ **Replies:** 7\ **Last updated:** [September 19, 2017, 7:22am UTC](https://discuss.elastic.co/t/grokparsefailure-when-field-is-no-longer-in-the-message/100882 "2017-09-19T07:22:27Z")
Hi I have a log that randomly shows an ip address at the end of the line. I have successfully created a filter to isolate the ip address but when the log does not produce the number I get a grokparsefailure. I really ne… --- ## [Logstash, Kafka & Elasticsearch](https://discuss.elastic.co/t/logstash-kafka-elasticsearch/99406)
**Author:** [@jogoinar10](https://discuss.elastic.co/u/jogoinar10)\ **Replies:** 11\ **Last updated:** [September 19, 2017, 6:42am UTC](https://discuss.elastic.co/t/logstash-kafka-elasticsearch/99406 "2017-09-19T06:42:18Z")
Is there a guide for configuration and setup for kafka and logstash connection. --- ## [Is it safe to install more recent Logstash Kafka plugin?](https://discuss.elastic.co/t/is-it-safe-to-install-more-recent-logstash-kafka-plugin/100823)
**Author:** [@kub1x](https://discuss.elastic.co/u/kub1x)\ **Replies:** 1\ **Last updated:** [September 19, 2017, 6:24am UTC](https://discuss.elastic.co/t/is-it-safe-to-install-more-recent-logstash-kafka-plugin/100823 "2017-09-19T06:24:33Z")
Hey, I see that logstash-input-kafka plugin is behind in the release than in the repo (currently v5.1.11 vs v8.0.2). I'd like to use it with kafka 0.11 which seems to be supported from v6.3.3. Is it safe to use? Thank… --- ## [Logstash-sflow-code output metic details?](https://discuss.elastic.co/t/logstash-sflow-code-output-metic-details/100962)
**Author:** [@lkpatel](https://discuss.elastic.co/u/lkpatel)\ **Replies:** 0\ **Last updated:** [September 19, 2017, 6:21am UTC](https://discuss.elastic.co/t/logstash-sflow-code-output-metic-details/100962 "2017-09-19T06:21:54Z")
Do logstash-sflow-codec output metric details like input\_octets or output\_octets are already multiplied by sampling rate ? [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=905) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=907)