# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=852 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 853 --- ## [Logstash 6.0 or 5.5.0 and Kafka 0.10.0.1](https://discuss.elastic.co/t/logstash-6-0-or-5-5-0-and-kafka-0-10-0-1/112635)
**Author:** [@moriol](https://discuss.elastic.co/u/moriol)\ **Replies:** 1\ **Last updated:** [December 21, 2017, 8:34am UTC](https://discuss.elastic.co/t/logstash-6-0-or-5-5-0-and-kafka-0-10-0-1/112635 "2017-12-21T08:34:37Z")
Hello, I am trying to integrate Logstash with Kafka. Logstash works very well with Kafka as an output, but when configured as input, it doesnt work. In logs it seems that communication between Kafka and Logstash works … --- ## [Logstash XML parsing problem](https://discuss.elastic.co/t/logstash-xml-parsing-problem/112678)
**Author:** [@lennart](https://discuss.elastic.co/u/lennart)\ **Replies:** 2\ **Last updated:** [December 21, 2017, 7:32am UTC](https://discuss.elastic.co/t/logstash-xml-parsing-problem/112678 "2017-12-21T07:32:27Z")
Hello! I've been trying to figure out how to do XML parsing in logstash for almost nine hours in total. There's not that much information about it and several sources are really old and I'm terrible at coming up with the… --- ## [Field turns into text in elasticsearch after being mapped as INT in grok and being mutated into integer](https://discuss.elastic.co/t/field-turns-into-text-in-elasticsearch-after-being-mapped-as-int-in-grok-and-being-mutated-into-integer/112381)
**Author:** [@yishain11](https://discuss.elastic.co/u/yishain11)\ **Replies:** 4\ **Last updated:** [December 21, 2017, 8:12am UTC](https://discuss.elastic.co/t/field-turns-into-text-in-elasticsearch-after-being-mapped-as-int-in-grok-and-being-mutated-into-integer/112381 "2017-12-21T08:12:38Z")
Hi everyone! I have logstash filter (grok) and I matched certain phrases as INT. When I open the mapping in kibana, I see that elasticsearch classified the INT as text. Why is that? How do I change it? I want to make n… --- ## [\_grokparsefailure in Kibana tag (Synology messages)](https://discuss.elastic.co/t/grokparsefailure-in-kibana-tag-synology-messages/111496)
**Author:** [@hispeed](https://discuss.elastic.co/u/hispeed)\ **Replies:** 7\ **Last updated:** [December 21, 2017, 7:18am UTC](https://discuss.elastic.co/t/grokparsefailure-in-kibana-tag-synology-messages/111496 "2017-12-21T07:18:01Z")
Hi, I recieve a \_grokparsefailure in Kibana tags but all the fields are splitted correct. How can I remove the grokparsefailure for those incomming messages? if \[logsource\] == "BackupStation" { mutate { … --- ## [Grok filter. How do I break up the message information?](https://discuss.elastic.co/t/grok-filter-how-do-i-break-up-the-message-information/110566)
**Author:** [@E\_Marshall](https://discuss.elastic.co/u/E_Marshall)\ **Replies:** 3\ **Last updated:** [December 21, 2017, 6:35am UTC](https://discuss.elastic.co/t/grok-filter-how-do-i-break-up-the-message-information/110566 "2017-12-21T06:35:29Z")
I created a filter to break apart our log files and am having the following issue. I'm not able to figure out how to save the parts of the "message" to their own field or tag or whatever you call it. I'm 3 days new to lo… --- ## [Using Python subprocess to run Logstash](https://discuss.elastic.co/t/using-python-subprocess-to-run-logstash/112536)
**Author:** [@druzmieres](https://discuss.elastic.co/u/druzmieres)\ **Replies:** 4\ **Last updated:** [December 20, 2017, 10:57am UTC](https://discuss.elastic.co/t/using-python-subprocess-to-run-logstash/112536 "2017-12-20T10:57:52Z")
Hi, I'm using Ubuntu 16.04.3 LTS and want Logstash to read a static file and stop after that. I read you can monitor the sincedb file to know when Logstash has reached EOF (here). I wrote a Python script to try to achi… --- ## [Rejecting mapping update to as the final mapping would have more than 1 type: \[log, doc\]](https://discuss.elastic.co/t/rejecting-mapping-update-to-as-the-final-mapping-would-have-more-than-1-type-log-doc/111866)
**Author:** [@kyle\_che](https://discuss.elastic.co/u/kyle_che)\ **Replies:** 14\ **Last updated:** [December 20, 2017, 8:02pm UTC](https://discuss.elastic.co/t/rejecting-mapping-update-to-as-the-final-mapping-would-have-more-than-1-type-log-doc/111866 "2017-12-20T20:02:14Z")
I'm trying to use logstash 6.0.1 and I'm getting the following error from filebeats 6.0.1. I understand the error but not sure how to correct this issue. Rejecting mapping update to \[filebeat-6.0.1-postgres-2017.11.29\]… --- ## [Elastic Search My SQL Integration via LogStash](https://discuss.elastic.co/t/elastic-search-my-sql-integration-via-logstash/112644)
**Author:** [@sg850r](https://discuss.elastic.co/u/sg850r)\ **Replies:** 4\ **Last updated:** [December 20, 2017, 7:30pm UTC](https://discuss.elastic.co/t/elastic-search-my-sql-integration-via-logstash/112644 "2017-12-20T19:30:07Z")
We are trying to use Elastic Search for search capabilities for our site. All data in in MySQL and we need this data copied to elastic search via LogStash.copied here is the conf file and log file . But when I run the lo… --- ## [Log-stash does not trigger new copied files](https://discuss.elastic.co/t/log-stash-does-not-trigger-new-copied-files/110872)
**Author:** [@Maysam](https://discuss.elastic.co/u/Maysam)\ **Replies:** 2\ **Last updated:** [December 18, 2017, 4:10pm UTC](https://discuss.elastic.co/t/log-stash-does-not-trigger-new-copied-files/110872 "2017-12-18T16:10:04Z")
Hello I have a box that receives files in a target folder . I have the following config: input { file { path =\> \[ "/usr/local/ogr/target/\*/\*Group.json"\] codec =\> json start\_position =\> "beginning" } } … --- ## [Dissector mapping, field found in event but it was empty](https://discuss.elastic.co/t/dissector-mapping-field-found-in-event-but-it-was-empty/108013)
**Author:** [@Badger](https://discuss.elastic.co/u/Badger)\ **Replies:** 5\ **Last updated:** [December 20, 2017, 4:33pm UTC](https://discuss.elastic.co/t/dissector-mapping-field-found-in-event-but-it-was-empty/108013 "2017-12-20T16:33:39Z")
Dissect does not like this user's useragent string. All the fields get parsed correctly, including useragent, but it still gets tagged with \_dissectfailure. \[2017-11-16T15:28:04,752\]\[WARN \]\[org.logstash.dissect.Dissecto… --- ## [Process date via in logstash](https://discuss.elastic.co/t/process-date-via-in-logstash/112665)
**Author:** [@moriol](https://discuss.elastic.co/u/moriol)\ **Replies:** 4\ **Last updated:** [December 20, 2017, 4:00pm UTC](https://discuss.elastic.co/t/process-date-via-in-logstash/112665 "2017-12-20T16:00:40Z")
I have a date in specific form, its as follows; "MyDate" =\> "\[20, 12, 17, 14, 31, 28, 2, 4\]", which is supposed to be 20/12/17 14:31 24ms What is the best way to format this in Logstash? I believe I can use a match, b… --- ## [Definition of values in Logstash](https://discuss.elastic.co/t/definition-of-values-in-logstash/112659)
**Author:** [@saad](https://discuss.elastic.co/u/saad)\ **Replies:** 0\ **Last updated:** [December 20, 2017, 3:00pm UTC](https://discuss.elastic.co/t/definition-of-values-in-logstash/112659 "2017-12-20T15:00:56Z")
Hi can some tell me what is the meaning of the events section (in, filtered, out) when I run x.x.x.x:9600/\_node/stats/events?pretty below is the result: { "host" : "LOGSTASHTEST", "version" : "6.0.1", "http\_addre… --- ## [Kafka input plugin](https://discuss.elastic.co/t/kafka-input-plugin/112657)
**Author:** [@zeenon](https://discuss.elastic.co/u/zeenon)\ **Replies:** 0\ **Last updated:** [December 20, 2017, 2:57pm UTC](https://discuss.elastic.co/t/kafka-input-plugin/112657 "2017-12-20T14:57:39Z")
Reading the latest documentation https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html, I do not understand the matrix. No mention of logstash 6.x, it seems a copy-paste of kibana 5.x ! So, is is po… --- ## [Logstash wmi input isssue](https://discuss.elastic.co/t/logstash-wmi-input-isssue/112650)
**Author:** [@Pachai\_muthu](https://discuss.elastic.co/u/Pachai_muthu)\ **Replies:** 0\ **Last updated:** [December 20, 2017, 2:13pm UTC](https://discuss.elastic.co/t/logstash-wmi-input-isssue/112650 "2017-12-20T14:13:24Z")
Hi team, I have installed logstash 6.0.0 version in centos 6.9 server and also installed logstash-wmi-input plugin 3.0.3 version. My Config file in /etc/logstash/conf.d/winsystem.conf input { wmi { query =\> "select … --- ## [Logstash defaults modules?](https://discuss.elastic.co/t/logstash-defaults-modules/112476)
**Author:** [@Kristoffer\_Fagerlund](https://discuss.elastic.co/u/Kristoffer_Fagerlund)\ **Replies:** 7\ **Last updated:** [December 20, 2017, 1:42pm UTC](https://discuss.elastic.co/t/logstash-defaults-modules/112476 "2017-12-20T13:42:14Z")
I just installed ELK and I got it to work for a few minutes then I booted the vm and now Im trying to get back. logstash doesnt listen to the port I specified in the conf file. I also noticed int the log file :/etc/log… --- ## [Losgtash running but there is no output](https://discuss.elastic.co/t/losgtash-running-but-there-is-no-output/112598)
**Author:** [@marouane](https://discuss.elastic.co/u/marouane)\ **Replies:** 3\ **Last updated:** [December 20, 2017, 1:13pm UTC](https://discuss.elastic.co/t/losgtash-running-but-there-is-no-output/112598 "2017-12-20T13:13:01Z")
Hello , i have an issue concerning logstash this is the log that i have, Settings: Default pipeline workers: 4 Pipeline main started my logstash log file input { http { type =\> "test\_http" host =\> "0.0.0.0" # de… --- ## [Logical expression in pipeline](https://discuss.elastic.co/t/logical-expression-in-pipeline/112621)
**Author:** [@bodi4](https://discuss.elastic.co/u/bodi4)\ **Replies:** 2\ **Last updated:** [December 20, 2017, 11:38am UTC](https://discuss.elastic.co/t/logical-expression-in-pipeline/112621 "2017-12-20T11:38:07Z")
Hi could someone explain logical expression in pipeline? The case is, for example Output section: output { if "C" in \[tags\] and \[filed\_A\] = A or "C" in \[tags\] and \[filed\_B\] = B elasticsearch {} } Second match grou… --- ## [GELF Output plugins adds a "protocol" field](https://discuss.elastic.co/t/gelf-output-plugins-adds-a-protocol-field/112625)
**Author:** [@ssiws](https://discuss.elastic.co/u/ssiws)\ **Replies:** 0\ **Last updated:** [December 20, 2017, 11:34am UTC](https://discuss.elastic.co/t/gelf-output-plugins-adds-a-protocol-field/112625 "2017-12-20T11:34:24Z")
Hello, I am using the GELF output plugin and I noticed that the GELF plugin seems to add a "protocol" field, always with 0 as value. Maybe there is an error in the code, here: https://github.com/logstash-plugins/logsta… --- ## [Aggregate filter - calculate difference between timestamps](https://discuss.elastic.co/t/aggregate-filter-calculate-difference-between-timestamps/111958)
**Author:** [@Jon\_D](https://discuss.elastic.co/u/Jon_D)\ **Replies:** 1\ **Last updated:** [December 20, 2017, 11:04am UTC](https://discuss.elastic.co/t/aggregate-filter-calculate-difference-between-timestamps/111958 "2017-12-20T11:04:20Z")
Hello, I'm using the following filters in my pipeline : filter { xml { id =\> "xmlparsing\_agg" source =\> "message" target =\> "doc" force\_array =\> false re… --- ## [Logstash ERROR: EADDRINUSE: Address already in use](https://discuss.elastic.co/t/logstash-error-eaddrinuse-address-already-in-use/112251)
**Author:** [@Bruno\_Dias](https://discuss.elastic.co/u/Bruno_Dias)\ **Replies:** 8\ **Last updated:** [December 20, 2017, 10:36am UTC](https://discuss.elastic.co/t/logstash-error-eaddrinuse-address-already-in-use/112251 "2017-12-20T10:36:29Z")
Hello, I'm trying to direct the pfsense logs to elasticsearch, all the tutorials I've found use the UDP port 5140, my pfsense can send the logs to that server on that port, but the logstash at startup shows the error bel… --- ## [Trying to separate fields from logs using GROK](https://discuss.elastic.co/t/trying-to-separate-fields-from-logs-using-grok/112375)
**Author:** [@Vivek\_Samaga](https://discuss.elastic.co/u/Vivek_Samaga)\ **Replies:** 4\ **Last updated:** [December 20, 2017, 10:34am UTC](https://discuss.elastic.co/t/trying-to-separate-fields-from-logs-using-grok/112375 "2017-12-20T10:34:25Z")
Hi am trying to separate fields from the log message below. 127.0.0.1 - - \[25/Sep/2017:06:13:23 -0500\] 4 "GET /CacheManager/HCD\_CLIENTS/BN\_HS HTTP/1.1" 200 118 \*\*\*:9090 "-" "-" I have been trying to parse / separate th… --- ## [Filtering XML input in Logstash](https://discuss.elastic.co/t/filtering-xml-input-in-logstash/112606)
**Author:** [@moriol](https://discuss.elastic.co/u/moriol)\ **Replies:** 0\ **Last updated:** [December 20, 2017, 10:27am UTC](https://discuss.elastic.co/t/filtering-xml-input-in-logstash/112606 "2017-12-20T10:27:10Z")
Hello, I have an XML file as an input in Logstash. I want to process and store only the header of the input, I dont care about the rest of the file. I have an input configured such as: file { path =\> "/home/test/file.… --- ## [Understanding running logstash container command](https://discuss.elastic.co/t/understanding-running-logstash-container-command/111923)
**Author:** [@marouane](https://discuss.elastic.co/u/marouane)\ **Replies:** 4\ **Last updated:** [December 20, 2017, 9:05am UTC](https://discuss.elastic.co/t/understanding-running-logstash-container-command/111923 "2017-12-20T09:05:21Z")
Hello every one, I'm working on creating the container logstash using docker run I found this commande and i want to understand what is means , docker run -d --name=logstash --restart=unless-stopped --link elasticsear… --- ## [Changing columns based on first character](https://discuss.elastic.co/t/changing-columns-based-on-first-character/112560)
**Author:** [@Errors](https://discuss.elastic.co/u/Errors)\ **Replies:** 4\ **Last updated:** [December 20, 2017, 9:03am UTC](https://discuss.elastic.co/t/changing-columns-based-on-first-character/112560 "2017-12-20T09:03:57Z")
I'm trying to pass a csv file which doesn't have column names. Each row in the file is also different (different variables, as well as varying row lengths), and would need more personalized column names. What I want to … --- ## [Cannot push to elasticsearch since migration of ES from 5.6 to 6.1](https://discuss.elastic.co/t/cannot-push-to-elasticsearch-since-migration-of-es-from-5-6-to-6-1/112446)
**Author:** [@jdefaver](https://discuss.elastic.co/u/jdefaver)\ **Replies:** 2\ **Last updated:** [December 20, 2017, 8:52am UTC](https://discuss.elastic.co/t/cannot-push-to-elasticsearch-since-migration-of-es-from-5-6-to-6-1/112446 "2017-12-20T08:52:29Z")
Hi, I have done a rolling migration from ES 5.6 to 6.1. I realized logstash could not push my events to ES anymore. I first updated logstash, then I realized I had the "\[include\_in\_all\] is not allowed for indices creat… --- ## [Update is not Working in Logstash JDBC Input](https://discuss.elastic.co/t/update-is-not-working-in-logstash-jdbc-input/112356)
**Author:** [@Sahul\_Hameed\_B](https://discuss.elastic.co/u/Sahul_Hameed_B)\ **Replies:** 5\ **Last updated:** [December 20, 2017, 8:36am UTC](https://discuss.elastic.co/t/update-is-not-working-in-logstash-jdbc-input/112356 "2017-12-20T08:36:17Z")
Hi I am working on Elastic Search using logstash conf file using sql query. I am connecting Sql Database using Logstash conf File. This is my Conf file input { jdbc { jdbc\_driver\_library =\> "D:\\Elastic\\elasticsearch-… --- ## [Change location of data and config folders (Windows)](https://discuss.elastic.co/t/change-location-of-data-and-config-folders-windows/112459)
**Author:** [@teejayuu](https://discuss.elastic.co/u/teejayuu)\ **Replies:** 1\ **Last updated:** [December 20, 2017, 8:15am UTC](https://discuss.elastic.co/t/change-location-of-data-and-config-folders-windows/112459 "2017-12-20T08:15:21Z")
How do I change the location of the data, logs and config folders in logstash from the default? Thanks --- ## [Duplicate query error](https://discuss.elastic.co/t/duplicate-query-error/112548)
**Author:** [@narasiman1986](https://discuss.elastic.co/u/narasiman1986)\ **Replies:** 1\ **Last updated:** [December 20, 2017, 5:43am UTC](https://discuss.elastic.co/t/duplicate-query-error/112548 "2017-12-20T05:43:54Z")
output { elasticsearch { hosts =\> \["localhost:9200"\] manage\_template =\> false index =\> "clinical" '\_id': \ } stdout { codec =\> rubydebug } } i want to remove the duplicates for that i written above but it… --- ## [Duplicate query problem](https://discuss.elastic.co/t/duplicate-query-problem/112542)
**Author:** [@narasiman1986](https://discuss.elastic.co/u/narasiman1986)\ **Replies:** 1\ **Last updated:** [December 20, 2017, 5:12am UTC](https://discuss.elastic.co/t/duplicate-query-problem/112542 "2017-12-20T05:12:08Z")
My conf file code as follows for removing records i written below code as follows input { jdbc { jdbc\_driver\_library =\> "D:\\mysql-connector-java-5.1.44\\mysql-connector-java-5.1.44\\mysql-connector-java-5.1.44-bin.jar"… --- ## [How to iterate over all fields with new Event API](https://discuss.elastic.co/t/how-to-iterate-over-all-fields-with-new-event-api/112514)
**Author:** [@Badger](https://discuss.elastic.co/u/Badger)\ **Replies:** 1\ **Last updated:** [December 20, 2017, 4:50am UTC](https://discuss.elastic.co/t/how-to-iterate-over-all-fields-with-new-event-api/112514 "2017-12-20T04:50:16Z")
With the new Event API that provides getter and setter and hides the hash, how do I iterate over all the fields in a ruby filter in logstash? Stuff like .each and .values appear not to work (not surprising if it is no l… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=851) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=853)