# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=972 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 973 --- ## [Need help in removing mysql field name for logstash elastic search json output](https://discuss.elastic.co/t/need-help-in-removing-mysql-field-name-for-logstash-elastic-search-json-output/87126)
**Author:** [@ashish\_pandey](https://discuss.elastic.co/u/ashish_pandey)\ **Replies:** 3\ **Last updated:** [June 2, 2017, 6:24am UTC](https://discuss.elastic.co/t/need-help-in-removing-mysql-field-name-for-logstash-elastic-search-json-output/87126 "2017-06-02T06:24:38Z")
Hi, I want to change the mysql fields id, project, key\_type, data, created\_at into a json without field name. Mysql Values are id: 1, project: slug1, key\_type: slug2, data: {"key1": "value1", "key2": "value2"}, created\_a… --- ## [Filebeat =\> Logstash Shipper=\>Kafka =\> logstash Parser =\> Elasticsearch =\>Kibana](https://discuss.elastic.co/t/filebeat-logstash-shipper-kafka-logstash-parser-elasticsearch-kibana/88007)
**Author:** [@Ravikumar\_G](https://discuss.elastic.co/u/Ravikumar_G)\ **Replies:** 0\ **Last updated:** [June 2, 2017, 2:09am UTC](https://discuss.elastic.co/t/filebeat-logstash-shipper-kafka-logstash-parser-elasticsearch-kibana/88007 "2017-06-02T02:09:49Z")
Hi I'm really confuse with following architecture pipeline My Goal is to send transport logs to filebeat =\> Logstash =\> Kafka =\> Logstash Parser =\> Elasticsearch =\> Kibana also I would like to set up all individual … --- ## [Taking too much time to index](https://discuss.elastic.co/t/taking-too-much-time-to-index/87918)
**Author:** [@munotshubham](https://discuss.elastic.co/u/munotshubham)\ **Replies:** 5\ **Last updated:** [June 2, 2017, 1:33am UTC](https://discuss.elastic.co/t/taking-too-much-time-to-index/87918 "2017-06-02T01:33:27Z")
So I'm trying to index a DB with 91 columns and around 300,000 rows. It is in a CSV file and I'm using logstash to load it in ES. It's been running since 20 hours, it still hasn't been indexed. For test purposes I had … --- ## [Elasticsearch output, strict mapping failure](https://discuss.elastic.co/t/elasticsearch-output-strict-mapping-failure/87990)
**Author:** [@unclethan](https://discuss.elastic.co/u/unclethan)\ **Replies:** 0\ **Last updated:** [June 1, 2017, 9:04pm UTC](https://discuss.elastic.co/t/elasticsearch-output-strict-mapping-failure/87990 "2017-06-01T21:04:16Z")
I have a logstash instance reading from a kafka topic, writing to an ES5.x cluster. When an ES insert fails due to "StrictDynamicMappingException", I would like to be alerted on the failure, optionally sending the paylo… --- ## [Getting metrics count of dynamic meter name](https://discuss.elastic.co/t/getting-metrics-count-of-dynamic-meter-name/87986)
**Author:** [@kbop543](https://discuss.elastic.co/u/kbop543)\ **Replies:** 0\ **Last updated:** [June 1, 2017, 8:52pm UTC](https://discuss.elastic.co/t/getting-metrics-count-of-dynamic-meter-name/87986 "2017-06-01T20:52:03Z")
Hey, So I have been testing how to use metrics to tell me the count of how many times a logfile logged errors. I use indexes to distinguish between logfiles by grabbing the filename from the path in a field called fil… --- ## [Filtering values with ruby in logstash 5.1](https://discuss.elastic.co/t/filtering-values-with-ruby-in-logstash-5-1/87982)
**Author:** [@harrigr](https://discuss.elastic.co/u/harrigr)\ **Replies:** 0\ **Last updated:** [June 1, 2017, 8:35pm UTC](https://discuss.elastic.co/t/filtering-values-with-ruby-in-logstash-5-1/87982 "2017-06-01T20:35:28Z")
So I have some data which is kv pairs separated by multiple characters, "::" to be precise. This issue: https://github.com/logstash-plugins/logstash-filter-kv/issues/15 shows that the built in kv filter cannot use multi-… --- ## [FAQ about the \*\*pattern\_definitions\*\* option in grok](https://discuss.elastic.co/t/faq-about-the-pattern-definitions-option-in-grok/87871)
**Author:** [@Fly](https://discuss.elastic.co/u/Fly)\ **Replies:** 1\ **Last updated:** [June 1, 2017, 8:24pm UTC](https://discuss.elastic.co/t/faq-about-the-pattern-definitions-option-in-grok/87871 "2017-06-01T20:24:24Z")
Hi all Can give me more description of pattern\_definitions option. When I read it in the configuration of the pipeline logstash with filebeat module. I do not yet understand its functionality in the configuration … --- ## [Getting lot of warnings "LogstashJSON - Failed action" in logstash config](https://discuss.elastic.co/t/getting-lot-of-warnings-logstashjson-failed-action-in-logstash-config/87974)
**Author:** [@rahul.dhammy](https://discuss.elastic.co/u/rahul.dhammy)\ **Replies:** 0\ **Last updated:** [June 1, 2017, 7:29pm UTC](https://discuss.elastic.co/t/getting-lot-of-warnings-logstashjson-failed-action-in-logstash-config/87974 "2017-06-01T19:29:31Z")
Hi I am getting lots of warnings in my logstash log. The warning says "LogstashJSON - Failed action" I somehow feel this is related to using my multiline codec. I am using logstash to parse my log files and put those in … --- ## [Dynamically assign xml target with xpath](https://discuss.elastic.co/t/dynamically-assign-xml-target-with-xpath/87963)
**Author:** [@timj123](https://discuss.elastic.co/u/timj123)\ **Replies:** 0\ **Last updated:** [June 1, 2017, 5:48pm UTC](https://discuss.elastic.co/t/dynamically-assign-xml-target-with-xpath/87963 "2017-06-01T17:48:58Z")
Is it possible to dynamically create an xml target based on the value of an xpath? I can't get below to work correctly. I would want the value of target to be the value of the xpath: /measInfo/@measInfoId", "TARGET … --- ## [JAVALOGMESSAGE takes it all](https://discuss.elastic.co/t/javalogmessage-takes-it-all/87780)
**Author:** [@remi128](https://discuss.elastic.co/u/remi128)\ **Replies:** 1\ **Last updated:** [June 1, 2017, 4:10pm UTC](https://discuss.elastic.co/t/javalogmessage-takes-it-all/87780 "2017-06-01T16:10:39Z")
i am starting to analyze logs with filebeat and logstash. Inside the grok definition i am using JAVALOGMESSAGE to extract the pure message. From my understanding the pattern "(.\*)" should mean something like "take any ch… --- ## [Timestamp - cant figure out](https://discuss.elastic.co/t/timestamp-cant-figure-out/87845)
**Author:** [@punit\_j](https://discuss.elastic.co/u/punit_j)\ **Replies:** 5\ **Last updated:** [June 1, 2017, 4:07pm UTC](https://discuss.elastic.co/t/timestamp-cant-figure-out/87845 "2017-06-01T16:07:56Z")
Hi , I have a web log entry like below: JINK|Information|0|05-17-2017 16:00:44:56|\[JNK\]|GET|/Services/portal/v1/CheckToken|End|System.Web.Http.Action|ReflectedHttpActionDescriptor.ExecuteAsync()|Action returned 'True'|… --- ## [Install logstash-input-fluentd plugin doesn't work (logstash 5.x)](https://discuss.elastic.co/t/install-logstash-input-fluentd-plugin-doesnt-work-logstash-5-x/87929)
**Author:** [@fragatina](https://discuss.elastic.co/u/fragatina)\ **Replies:** 0\ **Last updated:** [June 1, 2017, 2:44pm UTC](https://discuss.elastic.co/t/install-logstash-input-fluentd-plugin-doesnt-work-logstash-5-x/87929 "2017-06-01T14:44:37Z")
I have logstash 5.4 (latest release) installed through RPM. I cannot successfully install logstash-input-fluentd plugin with this version because it complains about: Bundler could not find compatible versions for gem … --- ## [How to stop duplicate entries using elasticsearch plugin](https://discuss.elastic.co/t/how-to-stop-duplicate-entries-using-elasticsearch-plugin/87880)
**Author:** [@Matthew\_Byrne](https://discuss.elastic.co/u/Matthew_Byrne)\ **Replies:** 9\ **Last updated:** [June 1, 2017, 1:44pm UTC](https://discuss.elastic.co/t/how-to-stop-duplicate-entries-using-elasticsearch-plugin/87880 "2017-06-01T13:44:00Z")
Morning all. Im trying to use the elasticsearch plugin to pull data from an existing elasticsearch instance. Both are running older versions of Elasticsearch, and thus im running an older version of logstash (1.5). I've … --- ## [Pull new enteries from distributed Cassandra database](https://discuss.elastic.co/t/pull-new-enteries-from-distributed-cassandra-database/87913)
**Author:** [@ayush1](https://discuss.elastic.co/u/ayush1)\ **Replies:** 0\ **Last updated:** [June 1, 2017, 1:21pm UTC](https://discuss.elastic.co/t/pull-new-enteries-from-distributed-cassandra-database/87913 "2017-06-01T13:21:59Z")
Hi all, I'm trying to setup Logstash to continuously pull information from Cassandra, in the same fashion it works with file inputs. I'm using the Cassandra JDBC Driver74 and I can successfully pull the contents. But l… --- ## [Unable to install x-pack in my logstash 5.4.0](https://discuss.elastic.co/t/unable-to-install-x-pack-in-my-logstash-5-4-0/87777)
**Author:** [@rodrigokroll](https://discuss.elastic.co/u/rodrigokroll)\ **Replies:** 2\ **Last updated:** [June 1, 2017, 1:18pm UTC](https://discuss.elastic.co/t/unable-to-install-x-pack-in-my-logstash-5-4-0/87777 "2017-06-01T13:18:48Z")
Hello guys, I need help. I'm trying to install the x-pack, but unfortunately without success. First, I've tried using the traditional way (online) and I had the following message. root@test:/opt/logstash/logstash-5… --- ## [Dynamic index name](https://discuss.elastic.co/t/dynamic-index-name/2490)
**Author:** [@maran](https://discuss.elastic.co/u/maran)\ **Replies:** 3\ **Last updated:** [June 1, 2017, 1:08pm UTC](https://discuss.elastic.co/t/dynamic-index-name/2490 "2017-06-01T13:08:43Z")
Hi! I'm trying to dynamically set index names with if-cases in my logstash.conf file. Basically if file name is A then indexname should be A. But for some reason my variable (type) is not set or accessible when I get t… --- ## [Regular Expression not working](https://discuss.elastic.co/t/regular-expression-not-working/87835)
**Author:** [@jigarpatel13533](https://discuss.elastic.co/u/jigarpatel13533)\ **Replies:** 5\ **Last updated:** [June 1, 2017, 1:04pm UTC](https://discuss.elastic.co/t/regular-expression-not-working/87835 "2017-06-01T13:04:08Z")
Hi, I am BEGINNER at Logstash 2.1. Trying to write config file for below log. LogFile: Timestamp Process TID Area Category … --- ## [How to stop logstash after first process](https://discuss.elastic.co/t/how-to-stop-logstash-after-first-process/87897)
**Author:** [@Sunil\_M](https://discuss.elastic.co/u/Sunil_M)\ **Replies:** 0\ **Last updated:** [June 1, 2017, 11:52am UTC](https://discuss.elastic.co/t/how-to-stop-logstash-after-first-process/87897 "2017-06-01T11:52:56Z")
Hi, I have a logstash conf whose input is elasticsearch and also output is to another elasticserach. I would like to read the input only once and stop the process so that it will not reinsert the same data to output el… --- ## [Logstash offline plugin installtion failed (logatsh-codec-protobuf)](https://discuss.elastic.co/t/logstash-offline-plugin-installtion-failed-logatsh-codec-protobuf/87887)
**Author:** [@sgarap](https://discuss.elastic.co/u/sgarap)\ **Replies:** 0\ **Last updated:** [June 1, 2017, 10:55am UTC](https://discuss.elastic.co/t/logstash-offline-plugin-installtion-failed-logatsh-codec-protobuf/87887 "2017-06-01T10:55:20Z")
Hi, I installed logstash-codec-protobuf on a staging server and exported zip file using offline-plugin-management. I tried to install the exported plugins zip file to logstash 5.2.0 on a different machine without inter… --- ## [Different multilines in one input](https://discuss.elastic.co/t/different-multilines-in-one-input/87683)
**Author:** [@JIELOGAN](https://discuss.elastic.co/u/JIELOGAN)\ **Replies:** 4\ **Last updated:** [June 1, 2017, 9:59am UTC](https://discuss.elastic.co/t/different-multilines-in-one-input/87683 "2017-06-01T09:59:46Z")
hi everyone,I have a problem here about multiline plugin,I have different log data in one file,and need use multiline to collapse multiline messages and merge them into a single event. my code is: input { file { … --- ## [Confused with running Logstash in docker](https://discuss.elastic.co/t/confused-with-running-logstash-in-docker/87752)
**Author:** [@Jaxon\_Kochel](https://discuss.elastic.co/u/Jaxon_Kochel)\ **Replies:** 7\ **Last updated:** [June 1, 2017, 9:06am UTC](https://discuss.elastic.co/t/confused-with-running-logstash-in-docker/87752 "2017-06-01T09:06:37Z")
Hello everyone, I'm looking for some instruction with how to run logstash in docker. I can get elasticsearch running okay but then im not sure how I'm supposed to start logstash. I'm also kinda confused about how I lin… --- ## [Grok not including trailing comma not working](https://discuss.elastic.co/t/grok-not-including-trailing-comma-not-working/87859)
**Author:** [@zanza54](https://discuss.elastic.co/u/zanza54)\ **Replies:** 0\ **Last updated:** [June 1, 2017, 8:10am UTC](https://discuss.elastic.co/t/grok-not-including-trailing-comma-not-working/87859 "2017-06-01T08:10:56Z")
Hey all I'm creating a filter that needs to filter a custom log message. The log message includes at the end the message of the log itself. This message is sometimes separated through comma's. I want to separate this… --- ## [Logstash not creating indexes](https://discuss.elastic.co/t/logstash-not-creating-indexes/87749)
**Author:** [@script007](https://discuss.elastic.co/u/script007)\ **Replies:** 2\ **Last updated:** [June 1, 2017, 7:10am UTC](https://discuss.elastic.co/t/logstash-not-creating-indexes/87749 "2017-06-01T07:10:08Z")
Hi team, i am new and trying to a new setup and trying to setup for snmptraps. ================== conf file: input { snmptrap { type =\> "traps" community =\> "public" host =\> "0.0.0.0" … --- ## [Getting 404 error on logstash start](https://discuss.elastic.co/t/getting-404-error-on-logstash-start/87740)
**Author:** [@gayathri](https://discuss.elastic.co/u/gayathri)\ **Replies:** 1\ **Last updated:** [June 1, 2017, 5:27am UTC](https://discuss.elastic.co/t/getting-404-error-on-logstash-start/87740 "2017-06-01T05:27:16Z")
Hi Team, We are running two logstash instances of different vesions on same server. One instance is working fine, and the other one throwing "404" error on start: failed action with response of 404, dropping action: … --- ## [Filebeats: ERR Connecting error publishing events](https://discuss.elastic.co/t/filebeats-err-connecting-error-publishing-events/87837)
**Author:** [@pace](https://discuss.elastic.co/u/pace)\ **Replies:** 1\ **Last updated:** [June 1, 2017, 5:19am UTC](https://discuss.elastic.co/t/filebeats-err-connecting-error-publishing-events/87837 "2017-06-01T05:19:48Z")
Hey all, So i seem to be getting an issue similar to others on here. Filebeats is running and so is logstash I can telnet to localhost:5044 Filebeat.yml #----------------------------- Logstash output ---------… --- ## [Combining two events in one](https://discuss.elastic.co/t/combining-two-events-in-one/43336)
**Author:** [@vsadokhin](https://discuss.elastic.co/u/vsadokhin)\ **Replies:** 8\ **Last updated:** [June 1, 2017, 1:23am UTC](https://discuss.elastic.co/t/combining-two-events-in-one/43336 "2017-06-01T01:23:40Z")
Hello over there! I'm looking for a way to combine two logs into one event. Say I have log lines that look like: \[timestamp\] \[unique id\] \[some message\] ... // different format/type log messages goes here \[timestamp\]… --- ## [Ruby LS filter error and slow logstash](https://discuss.elastic.co/t/ruby-ls-filter-error-and-slow-logstash/87806)
**Author:** [@ELK\_GB](https://discuss.elastic.co/u/ELK_GB)\ **Replies:** 0\ **Last updated:** [May 31, 2017, 9:00pm UTC](https://discuss.elastic.co/t/ruby-ls-filter-error-and-slow-logstash/87806 "2017-05-31T21:00:38Z")
Hi, I'm just wondering why whenever I try to use a ruby filter in my logstash conf file I get a very slow response from logstash when I try to restart it, plus the filter wont wont work and tailing LS logs with debug l… --- ## [Openshift and logstash](https://discuss.elastic.co/t/openshift-and-logstash/87776)
**Author:** [@pahermoso](https://discuss.elastic.co/u/pahermoso)\ **Replies:** 2\ **Last updated:** [May 31, 2017, 4:47pm UTC](https://discuss.elastic.co/t/openshift-and-logstash/87776 "2017-05-31T16:47:58Z")
Hi, I'm thinking to use logstash to increase the quality of logs coming from an Openshift platform. These logs are captured with FluentD and included into an Elasticsearch index directly. My intention is to handle log… --- ## [Logstash Aggregation](https://discuss.elastic.co/t/logstash-aggregation/87583)
**Author:** [@NounouGeek](https://discuss.elastic.co/u/NounouGeek)\ **Replies:** 5\ **Last updated:** [May 31, 2017, 4:18pm UTC](https://discuss.elastic.co/t/logstash-aggregation/87583 "2017-05-31T16:18:28Z")
Hello, I would like to know how to create an index from an existing Index with Elasticsearch aggregations. (In my case, i would like to get statistics per login: count\_connection, distinct\_count\_connection, first\_conn… --- ## [Grok URI extract](https://discuss.elastic.co/t/grok-uri-extract/54836)
**Author:** [@carrot](https://discuss.elastic.co/u/carrot)\ **Replies:** 6\ **Last updated:** [May 31, 2017, 3:54pm UTC](https://discuss.elastic.co/t/grok-uri-extract/54836 "2017-05-31T15:54:32Z")
I've been searching and reading documentation but I can't seem to find how to extract parameters from a URI. The best I could do so far is : grok { match =\> \[ "url", "%{URIPROTO:uri\_proto}://(?:%{USER:user}(?::\[… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=971) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=973)