# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1106 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1107 --- ## [Logstash Shield integration](https://discuss.elastic.co/t/logstash-shield-integration/59800)
**Author:** [@Kishore](https://discuss.elastic.co/u/Kishore)\ **Replies:** 7\ **Last updated:** [September 5, 2016, 3:12pm UTC](https://discuss.elastic.co/t/logstash-shield-integration/59800 "2016-09-05T15:12:21Z")
Hi , I have installed Shield on my Elasticsearch server, but unable to receive the events from logstash. Do i need to install shield plugin on logstash instance? And how the logstash output configuration looks like… --- ## [Logstash JDBC, checking for 0 records](https://discuss.elastic.co/t/logstash-jdbc-checking-for-0-records/59705)
**Author:** [@sombilat](https://discuss.elastic.co/u/sombilat)\ **Replies:** 3\ **Last updated:** [September 5, 2016, 1:33pm UTC](https://discuss.elastic.co/t/logstash-jdbc-checking-for-0-records/59705 "2016-09-05T13:33:32Z")
How do I check for JDBC if it returns 0 records? When I run JDBC, and it returns 0 records or has nothing to output. How do I say if the records retrieved are 0? --- ## [Logstash, split event from an xml file in multiples documents keeping information from root tags](https://discuss.elastic.co/t/logstash-split-event-from-an-xml-file-in-multiples-documents-keeping-information-from-root-tags/59801)
**Author:** [@Meenakshi07](https://discuss.elastic.co/u/Meenakshi07)\ **Replies:** 8\ **Last updated:** [September 5, 2016, 1:06pm UTC](https://discuss.elastic.co/t/logstash-split-event-from-an-xml-file-in-multiples-documents-keeping-information-from-root-tags/59801 "2016-09-05T13:06:58Z")
Parsing xml file via logstash. The documents should be parsed in a flattened structure. Example: \ \ \ \ \… --- ## [Date field to be loaded as @timestamp](https://discuss.elastic.co/t/date-field-to-be-loaded-as-timestamp/59795)
**Author:** [@bongo\_zg](https://discuss.elastic.co/u/bongo_zg)\ **Replies:** 4\ **Last updated:** [September 5, 2016, 12:42pm UTC](https://discuss.elastic.co/t/date-field-to-be-loaded-as-timestamp/59795 "2016-09-05T12:42:34Z")
I would like to load certain data but, when trying to create vertical bar chart with date value on x-axis (date histogram), I am unable to use it. The only field I can is @timestamp, and therefore would like to load it … --- ## [My own logstash filter doesn't extract the input field value correctly](https://discuss.elastic.co/t/my-own-logstash-filter-doesnt-extract-the-input-field-value-correctly/59803)
**Author:** [@sen](https://discuss.elastic.co/u/sen)\ **Replies:** 2\ **Last updated:** [September 5, 2016, 11:57am UTC](https://discuss.elastic.co/t/my-own-logstash-filter-doesnt-extract-the-input-field-value-correctly/59803 "2016-09-05T11:57:41Z")
Hello folks, I have built my own plugin filter for logstash. The principle of this filter is to get some coordinates (x,y) formated in a coordinates system called "lambertII" and to convert them in gps system (latitu… --- ## [Setting a filter that transforms "NULL" values to a default value from Mysql to elasticSearch](https://discuss.elastic.co/t/setting-a-filter-that-transforms-null-values-to-a-default-value-from-mysql-to-elasticsearch/59642)
**Author:** [@Sadok\_Mtir](https://discuss.elastic.co/u/Sadok_Mtir)\ **Replies:** 3\ **Last updated:** [September 5, 2016, 8:29am UTC](https://discuss.elastic.co/t/setting-a-filter-that-transforms-null-values-to-a-default-value-from-mysql-to-elasticsearch/59642 "2016-09-05T08:29:35Z")
The fields that I want to get them with default value could be NULL in MYSQL. This is my configuration for the logstash plugin. input { jdbc { jdbc\_connection\_string =\> "jdbc:mysql://localhost:3306/elements… --- ## [Logstash upstart service failed to start after logstash.log remove](https://discuss.elastic.co/t/logstash-upstart-service-failed-to-start-after-logstash-log-remove/59779)
**Author:** [@cazorla19](https://discuss.elastic.co/u/cazorla19)\ **Replies:** 2\ **Last updated:** [September 5, 2016, 8:27am UTC](https://discuss.elastic.co/t/logstash-upstart-service-failed-to-start-after-logstash-log-remove/59779 "2016-09-05T08:27:42Z")
The issue: facing with disk space deficiency, I remove in panic all stuffy log files. File logstash.log was there too. After further attempts to start service touching empty file logstash.log before. Service status becam… --- ## [Logstash 2.4.0 - redis input plugin issue?](https://discuss.elastic.co/t/logstash-2-4-0-redis-input-plugin-issue/59781)
**Author:** [@wmeens](https://discuss.elastic.co/u/wmeens)\ **Replies:** 0\ **Last updated:** [September 5, 2016, 6:43am UTC](https://discuss.elastic.co/t/logstash-2-4-0-redis-input-plugin-issue/59781 "2016-09-05T06:43:45Z")
Hi All, With a fresh 2.4.0 Logstash deployment I hit an issue with the Redis input plugin. While the plugin was getting loaded (shown in debug mode) nothing was getting picked up from the Redis (v 3.2.3) queue. Redis… --- ## [Incremental data on JDBC](https://discuss.elastic.co/t/incremental-data-on-jdbc/59776)
**Author:** [@Mormaii](https://discuss.elastic.co/u/Mormaii)\ **Replies:** 1\ **Last updated:** [September 5, 2016, 5:39am UTC](https://discuss.elastic.co/t/incremental-data-on-jdbc/59776 "2016-09-05T05:39:56Z")
I'm getting input from the JDBC plugin on logstash like this: SELECT epoch,action\_id,player\_id,world\_id,block\_id FROM prism\_data The data looks like this: { "epoch" =\> 1473046041, "action\_id" =\> 27, … --- ## [How Do I Verify Logstash is Loading Data?](https://discuss.elastic.co/t/how-do-i-verify-logstash-is-loading-data/57579)
**Author:** [@hisaotsu](https://discuss.elastic.co/u/hisaotsu)\ **Replies:** 5\ **Last updated:** [September 5, 2016, 1:00am UTC](https://discuss.elastic.co/t/how-do-i-verify-logstash-is-loading-data/57579 "2016-09-05T01:00:18Z")
I just set up Elastic Search, Kibana and Logstash. I am trying to load up the historical apache log to Elastic Search via Logstash. I have the below configuration for Logstash, however, it seems to stop after showing t… --- ## [Why do I still have a logstash-\* index? (newbie)](https://discuss.elastic.co/t/why-do-i-still-have-a-logstash-index-newbie/59689)
**Author:** [@jazzed](https://discuss.elastic.co/u/jazzed)\ **Replies:** 1\ **Last updated:** [September 4, 2016, 7:56pm UTC](https://discuss.elastic.co/t/why-do-i-still-have-a-logstash-index-newbie/59689 "2016-09-04T19:56:04Z")
I've configured filebeat to use prod-\* as the index. How come I still have a logstash-\* index that seems to contain different fields. Is it a default catch all index? --- ## [Restrict Logstash to map values to its specific Columns Using CSV filter](https://discuss.elastic.co/t/restrict-logstash-to-map-values-to-its-specific-columns-using-csv-filter/59303)
**Author:** [@karthikeyan95](https://discuss.elastic.co/u/karthikeyan95)\ **Replies:** 1\ **Last updated:** [September 4, 2016, 7:44pm UTC](https://discuss.elastic.co/t/restrict-logstash-to-map-values-to-its-specific-columns-using-csv-filter/59303 "2016-09-04T19:44:57Z")
Hi everyone, I am trying to store data from CSV files into Elasticsearch using Logstash. i have a logstash configuration using the csv filter containing columns say csv{ columns =\> \["A", "B", "C","D","E"\] seperator =\> … --- ## [Duplicate data in logstash](https://discuss.elastic.co/t/duplicate-data-in-logstash/59758)
**Author:** [@Mikail\_Land](https://discuss.elastic.co/u/Mikail_Land)\ **Replies:** 1\ **Last updated:** [September 4, 2016, 1:24pm UTC](https://discuss.elastic.co/t/duplicate-data-in-logstash/59758 "2016-09-04T13:24:39Z")
i have duplicate data in logstash my config file is : input { file { path =\> "/var/log/flask/access\*" type =\> "flask\_access" max\_open\_files =\> 409599 } stdin{} } filter { mutate { replace =\> { "typ… --- ## [Delete all documents from index before indexing new set](https://discuss.elastic.co/t/delete-all-documents-from-index-before-indexing-new-set/59735)
**Author:** [@sivabalan19](https://discuss.elastic.co/u/sivabalan19)\ **Replies:** 1\ **Last updated:** [September 3, 2016, 8:55pm UTC](https://discuss.elastic.co/t/delete-all-documents-from-index-before-indexing-new-set/59735 "2016-09-03T20:55:01Z")
Hi, My jdbc input plugin is getting all the new documents which needs to be indexed. I do have a requirement to delete all the documents before indexing the new set. Please suggest me how i can delete my old document… --- ## [Generating groc and save it to different variables](https://discuss.elastic.co/t/generating-groc-and-save-it-to-different-variables/59647)
**Author:** [@elasticheart](https://discuss.elastic.co/u/elasticheart)\ **Replies:** 1\ **Last updated:** [September 3, 2016, 7:46pm UTC](https://discuss.elastic.co/t/generating-groc-and-save-it-to-different-variables/59647 "2016-09-03T19:46:15Z")
Hi, I have a data like; \ \ \ \ \ \ \ \ **Author:** [@rmalired](https://discuss.elastic.co/u/rmalired)\ **Replies:** 2\ **Last updated:** [September 3, 2016, 4:17pm UTC](https://discuss.elastic.co/t/multiline-codec-in-input/59451 "2016-09-03T16:17:46Z") I have a scenario where filebeat send information from access.log as well as application.log. The application log can be multiline. so in the logstash.conf i have a multiline codec looking for a timestamp as the patte… --- ## [New user - Simple problem with adding a field with mutate](https://discuss.elastic.co/t/new-user-simple-problem-with-adding-a-field-with-mutate/59605)
**Author:** [@jazzed](https://discuss.elastic.co/u/jazzed)\ **Replies:** 3\ **Last updated:** [September 3, 2016, 3:36pm UTC](https://discuss.elastic.co/t/new-user-simple-problem-with-adding-a-field-with-mutate/59605 "2016-09-03T15:36:51Z")
I want to add a new field based on the content of the source field from filebeat. The filebeat-\* index shows SOME logs get the new field, and other logs are empty. However, the logstash-\* index has file defined for eve… --- ## [Http poller illegal argument exception](https://discuss.elastic.co/t/http-poller-illegal-argument-exception/58963)
**Author:** [@Crai](https://discuss.elastic.co/u/Crai)\ **Replies:** 3\ **Last updated:** [September 3, 2016, 6:27am UTC](https://discuss.elastic.co/t/http-poller-illegal-argument-exception/58963 "2016-09-03T06:27:09Z")
I am trying to use the http poller to pull eia.gov. U.S. Energy Information Administration - EIA - Independent Statistics and Analysis I'm doing the series query and it is a json output. It is attempting to pull the j… --- ## [Elastic Stack for 100 mn log lines with beat, logstah, elastic, kibana](https://discuss.elastic.co/t/elastic-stack-for-100-mn-log-lines-with-beat-logstah-elastic-kibana/59720)
**Author:** [@userguy](https://discuss.elastic.co/u/userguy)\ **Replies:** 0\ **Last updated:** [September 3, 2016, 1:57am UTC](https://discuss.elastic.co/t/elastic-stack-for-100-mn-log-lines-with-beat-logstah-elastic-kibana/59720 "2016-09-03T01:57:16Z")
Hello team, We are getting around 900gb of logs per per day and i want to process on realtime but in logstash i am getting circuit breaker as of now.. We have used 10 servers 4 for logstash and 6 for elasticsearch.. … --- ## [Logstash output and multiple destinations (elasticsearch and local file)](https://discuss.elastic.co/t/logstash-output-and-multiple-destinations-elasticsearch-and-local-file/58895)
**Author:** [@somerandomguy](https://discuss.elastic.co/u/somerandomguy)\ **Replies:** 9\ **Last updated:** [September 3, 2016, 1:19am UTC](https://discuss.elastic.co/t/logstash-output-and-multiple-destinations-elasticsearch-and-local-file/58895 "2016-09-03T01:19:54Z")
Hello. I have a requirement to send beats to multiple locations, ES (which is workning fine) and to a local file (that will be processed by another system). For the local file I might need to format it out of JSON format… --- ## [Logstash Index error : \[logstash-\*\] IndexNotFoundException\[no such index\]](https://discuss.elastic.co/t/logstash-index-error-logstash-indexnotfoundexception-no-such-index/37857)
**Author:** [@carmelom](https://discuss.elastic.co/u/carmelom)\ **Replies:** 27\ **Last updated:** [September 3, 2016, 1:17am UTC](https://discuss.elastic.co/t/logstash-index-error-logstash-indexnotfoundexception-no-such-index/37857 "2016-09-03T01:17:27Z")
Hello, I am new for ELK. I am using : - elasticsearch-2.1.0 - logstash-2.1.1 - kibana-4.3.0-windows I tried to configure ELK to monitoring my application logs and I followed different tutorials and different logsta… --- ## [Logstash 1.5.2-1 Service Stops after 15s](https://discuss.elastic.co/t/logstash-1-5-2-1-service-stops-after-15s/59713)
**Author:** [@ashraf1390](https://discuss.elastic.co/u/ashraf1390)\ **Replies:** 1\ **Last updated:** [September 2, 2016, 11:58pm UTC](https://discuss.elastic.co/t/logstash-1-5-2-1-service-stops-after-15s/59713 "2016-09-02T23:58:19Z")
Hey Everyone !!! Logstash service stops after 15s, kindly help... OS: CentOS 7 Installed rpms Elasticsearch 1.6.0 Kibana 4.1.1 Logstash 1.5.2-1 java-1.7.0-openjdk \[root@cenvm01 conf.d\]# ls 01-logstash-initial.con… --- ## [Using http\_poller to search Elasticsearch](https://discuss.elastic.co/t/using-http-poller-to-search-elasticsearch/59612)
**Author:** [@jerrac](https://discuss.elastic.co/u/jerrac)\ **Replies:** 3\ **Last updated:** [September 2, 2016, 9:39pm UTC](https://discuss.elastic.co/t/using-http-poller-to-search-elasticsearch/59612 "2016-09-02T21:39:35Z")
I want to use http\_poller to do the equivalent of this: curl -XGET 'http://esnode:9200/\_all/\_search?pretty' -d '{ "query": { "term": { "type" : "syslog" } } }' Basically, I want… --- ## [Logstash environment variables missing](https://discuss.elastic.co/t/logstash-environment-variables-missing/59616)
**Author:** [@jdambly](https://discuss.elastic.co/u/jdambly)\ **Replies:** 2\ **Last updated:** [September 2, 2016, 8:37pm UTC](https://discuss.elastic.co/t/logstash-environment-variables-missing/59616 "2016-09-02T20:37:03Z")
I have the following environment vars ansible@sv5enalysis01:~$ printenv | grep REDIS REDIS\_HOST=172.16.4.35 REDIS\_PORT=6381 ansible@sv5enalysis01:~$ but when I start logstash it doesn't replace them ansible@sv5e… --- ## [Logstash conf problem](https://discuss.elastic.co/t/logstash-conf-problem/47397)
**Author:** [@Jpuch](https://discuss.elastic.co/u/Jpuch)\ **Replies:** 4\ **Last updated:** [September 2, 2016, 8:30pm UTC](https://discuss.elastic.co/t/logstash-conf-problem/47397 "2016-09-02T20:30:10Z")
Hi, I have a problem with logstash, when i add a grok filter, logstash seem to stop sending data to ES or kibana. My data come by udp protocol on port 514 here is my filter conf file (the ELK system is working fin… --- ## [How to generate logstash-forwarder key and crt with TSL version 1.2](https://discuss.elastic.co/t/how-to-generate-logstash-forwarder-key-and-crt-with-tsl-version-1-2/59688)
**Author:** [@nagesh](https://discuss.elastic.co/u/nagesh)\ **Replies:** 0\ **Last updated:** [September 2, 2016, 4:13pm UTC](https://discuss.elastic.co/t/how-to-generate-logstash-forwarder-key-and-crt-with-tsl-version-1-2/59688 "2016-09-02T16:13:58Z")
Currently our Logstash server and client are getting communicated with version TSL version 1. We need to generate new cert and key files using TSL version 1.2 using openssl. Can some one suggest me how to pass TSL vers… --- ## [Can I attach a messages byte-count as a metadata field?](https://discuss.elastic.co/t/can-i-attach-a-messages-byte-count-as-a-metadata-field/59531)
**Author:** [@Ryan\_Grannell](https://discuss.elastic.co/u/Ryan_Grannell)\ **Replies:** 6\ **Last updated:** [September 2, 2016, 3:08pm UTC](https://discuss.elastic.co/t/can-i-attach-a-messages-byte-count-as-a-metadata-field/59531 "2016-09-02T15:08:56Z")
Hi; I'm currently running a Logstash server in which I receive messages from various sources. I'd like to find out the the approximate byte-count of each incoming message and attach this as a @metadata field, so I can dr… --- ## [Indexing custom fields from grok filter](https://discuss.elastic.co/t/indexing-custom-fields-from-grok-filter/59595)
**Author:** [@rsaeks](https://discuss.elastic.co/u/rsaeks)\ **Replies:** 5\ **Last updated:** [September 2, 2016, 2:42pm UTC](https://discuss.elastic.co/t/indexing-custom-fields-from-grok-filter/59595 "2016-09-02T14:42:14Z")
Good morning / afternoon / evening, I've created a custom grok pattern and ran through the pattern debugging for the message to make sure the named captures are pulling correctly. I'm currently defining the following: … --- ## [Summarizing Elasticsearch using Logstash](https://discuss.elastic.co/t/summarizing-elasticsearch-using-logstash/59669)
**Author:** [@rjsteckel](https://discuss.elastic.co/u/rjsteckel)\ **Replies:** 0\ **Last updated:** [September 2, 2016, 1:23pm UTC](https://discuss.elastic.co/t/summarizing-elasticsearch-using-logstash/59669 "2016-09-02T13:23:54Z")
We're using postgres as our data warehouse and elasticsearch as our UI's datastore. We then use logstash with jdbc inputs to periodically update elasticsearch from postgres. I would like to create a new index in elast… --- ## [Http poll - handling responses with both old and new events](https://discuss.elastic.co/t/http-poll-handling-responses-with-both-old-and-new-events/59664)
**Author:** [@Cyderic](https://discuss.elastic.co/u/Cyderic)\ **Replies:** 0\ **Last updated:** [September 2, 2016, 12:15pm UTC](https://discuss.elastic.co/t/http-poll-handling-responses-with-both-old-and-new-events/59664 "2016-09-02T12:15:16Z")
Hi folks, I would like to get some general ideas for the following challenge: We got a system that makes some checks that we want to grab with logstash. This system provides an simple API point which responds with al… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1105) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1107)