# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=648 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 649 --- ## [Geo\_point field extraction with Logstash](https://discuss.elastic.co/t/geo-point-field-extraction-with-logstash/156902)
**Author:** [@M.alsioufi](https://discuss.elastic.co/u/M.alsioufi)\ **Replies:** 0\ **Last updated:** [November 15, 2018, 4:00pm UTC](https://discuss.elastic.co/t/geo-point-field-extraction-with-logstash/156902 "2018-11-15T16:00:28Z")
I am inserting data from mysql to Elasticsearch using Logstash. Onw of the fields I am inserting is "geoCoordinates". This field is stored in Mysql in the format "lon;lat" I want to insert this field in my elasticsearch … --- ## [Unable to parse the apache error logs](https://discuss.elastic.co/t/unable-to-parse-the-apache-error-logs/156339)
**Author:** [@Nithani25](https://discuss.elastic.co/u/Nithani25)\ **Replies:** 1\ **Last updated:** [November 15, 2018, 3:38pm UTC](https://discuss.elastic.co/t/unable-to-parse-the-apache-error-logs/156339 "2018-11-15T15:38:53Z")
Hi Team, I am try to parse some set of apache access and error logs via filebeat input logs, although i am successfull in getting the apache logs in kibana, i am not getting any apache error messages in the kibana. I am… --- ## [How to aggregate heartbeat style log](https://discuss.elastic.co/t/how-to-aggregate-heartbeat-style-log/156139)
**Author:** [@Dypsok\_K](https://discuss.elastic.co/u/Dypsok_K)\ **Replies:** 1\ **Last updated:** [November 15, 2018, 12:29pm UTC](https://discuss.elastic.co/t/how-to-aggregate-heartbeat-style-log/156139 "2018-11-15T12:29:39Z")
Hi, I have some big files of log filled each 15 seconds with a heartbeat test result. For example one of this file is like this : 2018-07-17 10:14:41.646166 TRACE Service:Sales.API Result:OK 2018-07-17 10:14:56.… --- ## [Logstash/syslog/CEF 1024 character limit](https://discuss.elastic.co/t/logstash-syslog-cef-1024-character-limit/155614)
**Author:** [@nich](https://discuss.elastic.co/u/nich)\ **Replies:** 3\ **Last updated:** [November 15, 2018, 12:09pm UTC](https://discuss.elastic.co/t/logstash-syslog-cef-1024-character-limit/155614 "2018-11-15T12:09:22Z")
I have an issue with some devices sending CEF encoded TCP syslog messages to Logstash, where long messages (greater than 1024 characters) are split at the 1024 character position, with each section then being processed s… --- ## [Centralised Pipeline Management reload error](https://discuss.elastic.co/t/centralised-pipeline-management-reload-error/156842)
**Author:** [@programagor](https://discuss.elastic.co/u/programagor)\ **Replies:** 1\ **Last updated:** [November 15, 2018, 11:55am UTC](https://discuss.elastic.co/t/centralised-pipeline-management-reload-error/156842 "2018-11-15T11:55:30Z")
Greetings I just upgraded my ELK stack to 6.5.0, and 4 out of my 6 logstash nodes fail to load 1 out of 5 pipelines. The config is the same, and the pipeline is centrally managed, so I don't see why only some nodes are … --- ## [Error Installing Logstash Filter](https://discuss.elastic.co/t/error-installing-logstash-filter/156841)
**Author:** [@dawiro](https://discuss.elastic.co/u/dawiro)\ **Replies:** 0\ **Last updated:** [November 15, 2018, 11:32am UTC](https://discuss.elastic.co/t/error-installing-logstash-filter/156841 "2018-11-15T11:32:09Z")
Hi, I'm trying to install the de\_dot filter and am getting an error: An error occurred while installing logstash-core (5.6.3), and Bundler cannot continue. Make sure that \`gem install logstash-core -v '5.6.3'\` succeeds… --- ## [Logstash clone filter relies on deprecated \_type field](https://discuss.elastic.co/t/logstash-clone-filter-relies-on-deprecated-type-field/156553)
**Author:** [@jeffkirk1](https://discuss.elastic.co/u/jeffkirk1)\ **Replies:** 3\ **Last updated:** [November 15, 2018, 10:37am UTC](https://discuss.elastic.co/t/logstash-clone-filter-relies-on-deprecated-type-field/156553 "2018-11-15T10:37:06Z")
I just went through what appears to be a rite of passage for Logstash users and discovering that the "clone" filter won't actually do anything unless the "clones" array has at least one element. The result is a cloned e… --- ## [Logstash filter and output use only one CPU core](https://discuss.elastic.co/t/logstash-filter-and-output-use-only-one-cpu-core/155575)
**Author:** [@saisimo02](https://discuss.elastic.co/u/saisimo02)\ **Replies:** 11\ **Last updated:** [November 15, 2018, 9:04am UTC](https://discuss.elastic.co/t/logstash-filter-and-output-use-only-one-cpu-core/155575 "2018-11-15T09:04:28Z")
Hello, I am using losgtash to parse my xml files, however I can't increase the number of workers, when I use top -h only one worker is used ~98%. I am using 8 Go for jvm and 4 workers in the file logstash.yml logstash.… --- ## [Metrics filter plugin: does it work for delayed events? + general questions](https://discuss.elastic.co/t/metrics-filter-plugin-does-it-work-for-delayed-events-general-questions/156808)
**Author:** [@asp](https://discuss.elastic.co/u/asp)\ **Replies:** 0\ **Last updated:** [November 15, 2018, 8:28am UTC](https://discuss.elastic.co/t/metrics-filter-plugin-does-it-work-for-delayed-events-general-questions/156808 "2018-11-15T08:28:44Z")
Hi there, I have some questions about the metrics filter plugin. Does it respect the timestamp of the messages? Meaning, if I feed the metrics from different logs of different servers. One server for example has an i… --- ## [Logstash only insert fields in script](https://discuss.elastic.co/t/logstash-only-insert-fields-in-script/156803)
**Author:** [@M.alsioufi](https://discuss.elastic.co/u/M.alsioufi)\ **Replies:** 0\ **Last updated:** [November 15, 2018, 8:09am UTC](https://discuss.elastic.co/t/logstash-only-insert-fields-in-script/156803 "2018-11-15T08:09:44Z")
hi I am using Logstash to query MySQL database and insert its data to Elasticsearch cluster. In my Elasticsearch output plugin I use a script like the following script =\> 'if (ctx.\_source.nameslist== null){ ctx.\_sour… --- ## [Logstash upgraded without Filebeat and Winlogbeat upgrading](https://discuss.elastic.co/t/logstash-upgraded-without-filebeat-and-winlogbeat-upgrading/156799)
**Author:** [@xiaozhuqq48](https://discuss.elastic.co/u/xiaozhuqq48)\ **Replies:** 0\ **Last updated:** [November 15, 2018, 7:40am UTC](https://discuss.elastic.co/t/logstash-upgraded-without-filebeat-and-winlogbeat-upgrading/156799 "2018-11-15T07:40:33Z")
I upgrade my logstash version to 6.4.2, with logstash -input-beats plugin version 5.1.2. But my filebeats are still in version 5.1.2. It's not possible to upgrade in a short time because there are so many of them. Ca… --- ## [LogStash::Instrument::MetricStore::MetricNotFound](https://discuss.elastic.co/t/logstash-metricnotfound/156796)
**Author:** [@antianchi](https://discuss.elastic.co/u/antianchi)\ **Replies:** 0\ **Last updated:** [November 15, 2018, 6:59am UTC](https://discuss.elastic.co/t/logstash-metricnotfound/156796 "2018-11-15T06:59:29Z")
Continuing the discussion from Logstash again is failing to create monitoring event: My problem: \[ERROR\]\[logstash.inputs.metrics \] Failed to create monitoring event {:message=\>"For path: events. Map keys: \[:pipelines,… --- ## [Logstash authentication to ES](https://discuss.elastic.co/t/logstash-authentication-to-es/156746)
**Author:** [@Mark\_Andre](https://discuss.elastic.co/u/Mark_Andre)\ **Replies:** 1\ **Last updated:** [November 15, 2018, 5:34am UTC](https://discuss.elastic.co/t/logstash-authentication-to-es/156746 "2018-11-15T05:34:42Z")
Hi all, I'm trying to determine if there's a way to send log data via logstash pipelines into Elastic without having to use saved passwords in the pipeline config. Is this possible? --- ## [Logstash cannot find filter Alter](https://discuss.elastic.co/t/logstash-cannot-find-filter-alter/156558)
**Author:** [@motts](https://discuss.elastic.co/u/motts)\ **Replies:** 3\ **Last updated:** [November 14, 2018, 7:05pm UTC](https://discuss.elastic.co/t/logstash-cannot-find-filter-alter/156558 "2018-11-14T19:05:07Z")
Hello, I am using Logstash 6.4 and I am trying to install the Alter plugin. I navigated to "/usr/share/logstash/bin" and then ran ./logstash-plugin install logstash-filter-alter Everything seemed to go OK. It found t… --- ## [How to capture XML files in email attachment using IMAP - for DMARC Reporting](https://discuss.elastic.co/t/how-to-capture-xml-files-in-email-attachment-using-imap-for-dmarc-reporting/156677)
**Author:** [@guibarati](https://discuss.elastic.co/u/guibarati)\ **Replies:** 1\ **Last updated:** [November 14, 2018, 5:23pm UTC](https://discuss.elastic.co/t/how-to-capture-xml-files-in-email-attachment-using-imap-for-dmarc-reporting/156677 "2018-11-14T17:23:47Z")
Is there a way to connect to an email account, download all emails, retrieve all the attachment files that are XML, then parse the XML and have the XML data populate the fields and discard the attachment? I tried using … --- ## [NGINX log parse issue](https://discuss.elastic.co/t/nginx-log-parse-issue/154245)
**Author:** [@rpendela](https://discuss.elastic.co/u/rpendela)\ **Replies:** 5\ **Last updated:** [November 14, 2018, 5:18pm UTC](https://discuss.elastic.co/t/nginx-log-parse-issue/154245 "2018-11-14T17:18:33Z")
Hello community, I am trying parse my custom NGINX logs and in the kibana Debugger I can see parsing is correct but when I apply in the pipeline I an seeing \_grokoparsefailure with parsed log. I am not sure why it's fai… --- ## [Logstash Json Error](https://discuss.elastic.co/t/logstash-json-error/156720)
**Author:** [@Rolf\_Nufable](https://discuss.elastic.co/u/Rolf_Nufable)\ **Replies:** 0\ **Last updated:** [November 14, 2018, 5:14pm UTC](https://discuss.elastic.co/t/logstash-json-error/156720 "2018-11-14T17:14:05Z")
Hi All, I'm fairly new to logstash and ELK stack and I'm stuck in trying to ingest the json file from a filebeat instance. I can receive the json entries from filebeat just fine but all values/fields are in the message… --- ## [Indexing Child Document Using Logstash (Elastic v6.x)](https://discuss.elastic.co/t/indexing-child-document-using-logstash-elastic-v6-x/156706)
**Author:** [@Panda8](https://discuss.elastic.co/u/Panda8)\ **Replies:** 0\ **Last updated:** [November 14, 2018, 4:18pm UTC](https://discuss.elastic.co/t/indexing-child-document-using-logstash-elastic-v6-x/156706 "2018-11-14T16:18:42Z")
Hi Now Since Elasticsearch (v 6.x) has removed '\_parent' and introduced 'joinfields' for specifying parent child relationship. How to index child document in elasticsearch ( having join fields specifying the parent chil… --- ## [Case insensitive regex match with custom grok pattern](https://discuss.elastic.co/t/case-insensitive-regex-match-with-custom-grok-pattern/154280)
**Author:** [@YvorL](https://discuss.elastic.co/u/YvorL)\ **Replies:** 1\ **Last updated:** [November 14, 2018, 4:12pm UTC](https://discuss.elastic.co/t/case-insensitive-regex-match-with-custom-grok-pattern/154280 "2018-11-14T16:12:06Z")
Hi! I have a log with several fields. From those, I have one quoted string which needs to be matched against a regular expression pattern (case insensitive) and depending on the result I need to add a new field while ke… --- ## [One Logstash server not pushing docs through -other is fine](https://discuss.elastic.co/t/one-logstash-server-not-pushing-docs-through-other-is-fine/156176)
**Author:** [@fionamacd](https://discuss.elastic.co/u/fionamacd)\ **Replies:** 1\ **Last updated:** [November 14, 2018, 4:09pm UTC](https://discuss.elastic.co/t/one-logstash-server-not-pushing-docs-through-other-is-fine/156176 "2018-11-14T16:09:57Z")
Hi, Looking for some suggestion on how to investigate this issue really. I have one logstash server set up in Ubuntu (6.4.2) and I have a slew of servers sending filebeat, metricbeat and winlogbeat logs to this server.… --- ## [Append values to array field](https://discuss.elastic.co/t/append-values-to-array-field/156476)
**Author:** [@M.alsioufi](https://discuss.elastic.co/u/M.alsioufi)\ **Replies:** 2\ **Last updated:** [November 14, 2018, 2:45pm UTC](https://discuss.elastic.co/t/append-values-to-array-field/156476 "2018-11-14T14:45:36Z")
Hi, I am running Logstash to insert data from MySQL database to Elasticsearch. In MySQL database I have one-to-many relationship so one value from table A has zero or more values in table B. I have made the configurat… --- ## [Logstash cloudwatch input plugn](https://discuss.elastic.co/t/logstash-cloudwatch-input-plugn/156687)
**Author:** [@shahid](https://discuss.elastic.co/u/shahid)\ **Replies:** 0\ **Last updated:** [November 14, 2018, 2:00pm UTC](https://discuss.elastic.co/t/logstash-cloudwatch-input-plugn/156687 "2018-11-14T14:00:55Z")
Hi team , I install the logstash -cloudwatch input plugin on my ELK server and inside the /logstash/config.d/cloudwatch.conf here my setting i put inside the con file(cloudwatch.conf). \< input { beats { port =\> "5… --- ## [Multiple logs with errors in Kibana/Discover view](https://discuss.elastic.co/t/multiple-logs-with-errors-in-kibana-discover-view/156671)
**Author:** [@joanna\_a](https://discuss.elastic.co/u/joanna_a)\ **Replies:** 0\ **Last updated:** [November 14, 2018, 1:26pm UTC](https://discuss.elastic.co/t/multiple-logs-with-errors-in-kibana-discover-view/156671 "2018-11-14T13:26:55Z")
Hi, I have an issue with multiplying info about error message. This is a filter which works (I guess it works correctly, because all errors, warning were catch properly): filter { grok { match =\> { "message" =\> '%{NU… --- ## [Mapper parsing exception - \[variables.variable\] tried to parse field \[variable\] as object, but found a concrete value](https://discuss.elastic.co/t/mapper-parsing-exception-variables-variable-tried-to-parse-field-variable-as-object-but-found-a-concrete-value/156656)
**Author:** [@jeisonsj](https://discuss.elastic.co/u/jeisonsj)\ **Replies:** 0\ **Last updated:** [November 14, 2018, 12:31pm UTC](https://discuss.elastic.co/t/mapper-parsing-exception-variables-variable-tried-to-parse-field-variable-as-object-but-found-a-concrete-value/156656 "2018-11-14T12:31:00Z")
Hi guys, I'm facing an error only when using lowercase word "variables" for parsing a JSON event, that comes from Kafka, in a Filter clause. I tested using the uppercase and no errors occurs. But I would like to unders… --- ## [Enable Monitoring only on Logstash](https://discuss.elastic.co/t/enable-monitoring-only-on-logstash/156649)
**Author:** [@filipe.ferreira](https://discuss.elastic.co/u/filipe.ferreira)\ **Replies:** 0\ **Last updated:** [November 14, 2018, 11:57am UTC](https://discuss.elastic.co/t/enable-monitoring-only-on-logstash/156649 "2018-11-14T11:57:42Z")
Hi, is it possible to enable monitoring only for logstash? Best Regards, Filipe Ferreira. --- ## [Filter for Sidewinder syslog](https://discuss.elastic.co/t/filter-for-sidewinder-syslog/156598)
**Author:** [@mortueta](https://discuss.elastic.co/u/mortueta)\ **Replies:** 1\ **Last updated:** [November 14, 2018, 11:49am UTC](https://discuss.elastic.co/t/filter-for-sidewinder-syslog/156598 "2018-11-14T11:49:07Z")
Hi all, I am trying to figure out which is the best way to filter a log of a sidewinnder firewall. The situation is that depending the log type the message contains more or less fields. The pattern is that always we hav… --- ## [Can't even parse very simple data and get \_grokparsefailure as a tag](https://discuss.elastic.co/t/cant-even-parse-very-simple-data-and-get-grokparsefailure-as-a-tag/156437)
**Author:** [@Mustapha\_MJ](https://discuss.elastic.co/u/Mustapha_MJ)\ **Replies:** 18\ **Last updated:** [November 14, 2018, 11:30am UTC](https://discuss.elastic.co/t/cant-even-parse-very-simple-data-and-get-grokparsefailure-as-a-tag/156437 "2018-11-14T11:30:24Z")
i want to get started with ELK but i'm stuck every-time, after 2 weeks of complication i can finally ingest my sample log into elastic search, but fell into another problem, i can't see all my records in kibana and those… --- ## [Messages from gelf input not ending-up in ES](https://discuss.elastic.co/t/messages-from-gelf-input-not-ending-up-in-es/156291)
**Author:** [@plex1030](https://discuss.elastic.co/u/plex1030)\ **Replies:** 12\ **Last updated:** [November 14, 2018, 10:17am UTC](https://discuss.elastic.co/t/messages-from-gelf-input-not-ending-up-in-es/156291 "2018-11-14T10:17:02Z")
Hi, I was hoping to use the gelf input plugin to receive messages and put them into ES. This is my config: input { gelf { port =\> 12201 type =\> gelf host =\> "0.0.0.0" } } ... output { file { … --- ## [How to split Grok Message](https://discuss.elastic.co/t/how-to-split-grok-message/156592)
**Author:** [@Sripal](https://discuss.elastic.co/u/Sripal)\ **Replies:** 3\ **Last updated:** [November 14, 2018, 10:08am UTC](https://discuss.elastic.co/t/how-to-split-grok-message/156592 "2018-11-14T10:08:55Z")
Hi All, I'm trying to read the access log using gork filter. but i am unable to split the message. can anybody help. Here is my config file and sample output Thanks in advance . Sripal --- ## [Chown: filebeat.yml: operation not permitted](https://discuss.elastic.co/t/chown-filebeat-yml-operation-not-permitted/156624)
**Author:** [@tripstore](https://discuss.elastic.co/u/tripstore)\ **Replies:** 0\ **Last updated:** [November 14, 2018, 9:54am UTC](https://discuss.elastic.co/t/chown-filebeat-yml-operation-not-permitted/156624 "2018-11-14T09:54:10Z")
Hi, I started learning how to code two months ago, so everything is very new to me. Currently I'm trying to learn how to use logstash from the elastic website (learning how to move from mysql to elasticsearch using logst… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=647) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=649)