# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1009 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1010 --- ## [Logstash pipeline not starting](https://discuss.elastic.co/t/logstash-pipeline-not-starting/79937)
**Author:** [@g.le](https://discuss.elastic.co/u/g.le)\ **Replies:** 10\ **Last updated:** [March 29, 2017, 8:01pm UTC](https://discuss.elastic.co/t/logstash-pipeline-not-starting/79937 "2017-03-29T20:01:38Z")
Hello, I have created a rather complex Logstash config using various plugins. I have noticed that Logstash normally produces the following entries in /var/log/logstash/logstash-plain.log: \[2017-03-24T20:01:05,639\]\[IN… --- ## [Short\_message vs Long\_message](https://discuss.elastic.co/t/short-message-vs-long-message/80336)
**Author:** [@anshul\_pandey](https://discuss.elastic.co/u/anshul_pandey)\ **Replies:** 1\ **Last updated:** [March 29, 2017, 6:24pm UTC](https://discuss.elastic.co/t/short-message-vs-long-message/80336 "2017-03-29T18:24:05Z")
hi. I have a rabbitmq-\>Logstash-\>elasticSearch pipeline .When i look at the data being sent to elastic search, i see that some times, there is a short\_message and long\_message while other times there is just short\_Mess… --- ## [Optional \ on start of multiline pattern](https://discuss.elastic.co/t/optional-feff-on-start-of-multiline-pattern/80387)
**Author:** [@mhooper](https://discuss.elastic.co/u/mhooper)\ **Replies:** 0\ **Last updated:** [March 28, 2017, 11:14pm UTC](https://discuss.elastic.co/t/optional-feff-on-start-of-multiline-pattern/80387 "2017-03-28T23:14:41Z")
\< f e f f \>\[20170328, I have log lines coming from an application that start normally with "\[YYYYMMdd," which is easy to put into a multiline pattern, but occasionally there is a \< f e f f \> at the front of line. I … --- ## [Why I am getting a double mapping for my keyword fields?](https://discuss.elastic.co/t/why-i-am-getting-a-double-mapping-for-my-keyword-fields/80362)
**Author:** [@Nestor\_Manrique](https://discuss.elastic.co/u/Nestor_Manrique)\ **Replies:** 5\ **Last updated:** [March 29, 2017, 4:25pm UTC](https://discuss.elastic.co/t/why-i-am-getting-a-double-mapping-for-my-keyword-fields/80362 "2017-03-29T16:25:38Z")
Hello, I'm using http\_poller to bring some info from an API to ES. My output config looks like this. output { if \[type\] == 'ServerPool'{ elasticsearch { hosts =\> "localhost:9200" … --- ## [Filebeat & logstash Grok patterns](https://discuss.elastic.co/t/filebeat-logstash-grok-patterns/80340)
**Author:** [@Ravikumar\_G](https://discuss.elastic.co/u/Ravikumar_G)\ **Replies:** 4\ **Last updated:** [March 29, 2017, 3:24pm UTC](https://discuss.elastic.co/t/filebeat-logstash-grok-patterns/80340 "2017-03-29T15:24:54Z")
Hi I went through all the documents available in online have I'm sure I'm doing small mistake some were I'm unable to see follow log in kibana dashboard I'm transporting via filebeat to logstash else if "drive… --- ## [Create Custom geoip database for Logstash 5.2](https://discuss.elastic.co/t/create-custom-geoip-database-for-logstash-5-2/79473)
**Author:** [@Grenouille06](https://discuss.elastic.co/u/Grenouille06)\ **Replies:** 20\ **Last updated:** [March 29, 2017, 1:53pm UTC](https://discuss.elastic.co/t/create-custom-geoip-database-for-logstash-5-2/79473 "2017-03-29T13:53:17Z")
Hi, in old versions of Logstash, it was easy to create a custom dat file from a csv for having a geoip private ip database. The csv looks like : startIpNum,endIpNum,country,region,city,postalCode,latitude,longitude,me… --- ## [Using googlemaps geocoding API for address-\>geopoint](https://discuss.elastic.co/t/using-googlemaps-geocoding-api-for-address-geopoint/61555)
**Author:** [@Meenal.Luktuke](https://discuss.elastic.co/u/Meenal.Luktuke)\ **Replies:** 3\ **Last updated:** [March 29, 2017, 1:15pm UTC](https://discuss.elastic.co/t/using-googlemaps-geocoding-api-for-address-geopoint/61555 "2017-03-29T13:15:38Z")
Hi, I have a dataset which contains address, city etc. Using google's geocoding API in runtime, I would like to get geo-coordinates so that I can plot this on tilemap. But how to invoke this command in Logstash? Fo… --- ## [Jdbc input put data in all indexes](https://discuss.elastic.co/t/jdbc-input-put-data-in-all-indexes/80446)
**Author:** [@Amilyso](https://discuss.elastic.co/u/Amilyso)\ **Replies:** 6\ **Last updated:** [March 29, 2017, 1:00pm UTC](https://discuss.elastic.co/t/jdbc-input-put-data-in-all-indexes/80446 "2017-03-29T13:00:19Z")
Hi all, When I create a conf file with jdbc input, the data go to all my indexes. For example I did two conf files with jdbc input : input { jdbc { jdbc\_connection\_string =\> "jdbc:postgresql://localhost:5432/postgr… --- ## [Logstash grok customized filter f5 big-ip](https://discuss.elastic.co/t/logstash-grok-customized-filter-f5-big-ip/79240)
**Author:** [@mostafa](https://discuss.elastic.co/u/mostafa)\ **Replies:** 5\ **Last updated:** [March 29, 2017, 11:49am UTC](https://discuss.elastic.co/t/logstash-grok-customized-filter-f5-big-ip/79240 "2017-03-29T11:49:57Z")
Dear All, I installed ELK to use it to log F5 big ip , it is receiving logs successfully and I can search within logs in kibana,search is working very fast compared to any syslog I worked with it. however, I cannot pa… --- ## [Managing Logstash - restating on fail](https://discuss.elastic.co/t/managing-logstash-restating-on-fail/80471)
**Author:** [@jsps](https://discuss.elastic.co/u/jsps)\ **Replies:** 0\ **Last updated:** [March 29, 2017, 11:39am UTC](https://discuss.elastic.co/t/managing-logstash-restating-on-fail/80471 "2017-03-29T11:39:37Z")
Hi All, first time we are starting to use logstash. What is the best practice for restarting the process when it dies from lack of memory or network issue for example ? We are consuming from databases using the JDBC plu… --- ## [Export elasticsearch data into csv file](https://discuss.elastic.co/t/export-elasticsearch-data-into-csv-file/79978)
**Author:** [@Nikparab](https://discuss.elastic.co/u/Nikparab)\ **Replies:** 8\ **Last updated:** [March 29, 2017, 10:16am UTC](https://discuss.elastic.co/t/export-elasticsearch-data-into-csv-file/79978 "2017-03-29T10:16:57Z")
Hi I want to export data which is indexed in elasticsearch Here is my config file input { elasticsearch { hosts =\> \["localhost:9200"\] index =\> "mongolog1" query =\> ' { "query": { "bool": { "… --- ## [Best way to measure performance and efficiency](https://discuss.elastic.co/t/best-way-to-measure-performance-and-efficiency/80439)
**Author:** [@parrot](https://discuss.elastic.co/u/parrot)\ **Replies:** 2\ **Last updated:** [March 29, 2017, 8:45am UTC](https://discuss.elastic.co/t/best-way-to-measure-performance-and-efficiency/80439 "2017-03-29T08:45:46Z")
Hello, Because im parsing big amount of logs, i want to reduce the time it takes to parse them with logstash. Most of the configuration im using grok, so i want to compare it in different ways on different log files s… --- ## [Setup, configure Logstash for collecting peromance issues for MongoDB](https://discuss.elastic.co/t/setup-configure-logstash-for-collecting-peromance-issues-for-mongodb/80436)
**Author:** [@karunakarkotha](https://discuss.elastic.co/u/karunakarkotha)\ **Replies:** 0\ **Last updated:** [March 29, 2017, 8:23am UTC](https://discuss.elastic.co/t/setup-configure-logstash-for-collecting-peromance-issues-for-mongodb/80436 "2017-03-29T08:23:31Z")
I want to capture the MongoDB performance issue for MongoDB using Logstash, can anyone please provide me sameple config file and configuration steps --- ## [Unable to query ES](https://discuss.elastic.co/t/unable-to-query-es/80325)
**Author:** [@E.Eric](https://discuss.elastic.co/u/E.Eric)\ **Replies:** 0\ **Last updated:** [March 28, 2017, 4:24pm UTC](https://discuss.elastic.co/t/unable-to-query-es/80325 "2017-03-28T16:24:17Z")
Hi, here my config: \`elasticsearch { hosts =\> "localhost:9200" index=\>"servers" query =\> '{ "query": {"match": { "server\_id": "584" }} }' }\` It generates this error: ,"reason":"Cannot parse… --- ## [DNS resolving in JSON document](https://discuss.elastic.co/t/dns-resolving-in-json-document/79016)
**Author:** [@karthickv](https://discuss.elastic.co/u/karthickv)\ **Replies:** 19\ **Last updated:** [March 29, 2017, 7:37am UTC](https://discuss.elastic.co/t/dns-resolving-in-json-document/79016 "2017-03-29T07:37:03Z")
I have a nested JSON document which i'm pushing it to Elastic via Logstash. To resolve the ip address i have the DNS plugin installed. It works when i format the json as a single row & insert. But the time required to c… --- ## [Connect ssh and access log in logstash](https://discuss.elastic.co/t/connect-ssh-and-access-log-in-logstash/80406)
**Author:** [@manopmk](https://discuss.elastic.co/u/manopmk)\ **Replies:** 10\ **Last updated:** [March 29, 2017, 7:17am UTC](https://discuss.elastic.co/t/connect-ssh-and-access-log-in-logstash/80406 "2017-03-29T07:17:15Z")
Hi Buddy :slight\_smile: My log is in ssh which have username password port I have ELK pack in my local i want to connect ssh and access log for that I dont know how to configure logstash.conf can you help --- ## [Syslog additional fields](https://discuss.elastic.co/t/syslog-additional-fields/36389)
**Author:** [@theo.bot](https://discuss.elastic.co/u/theo.bot)\ **Replies:** 7\ **Last updated:** [March 29, 2017, 7:06am UTC](https://discuss.elastic.co/t/syslog-additional-fields/36389 "2017-03-29T07:06:35Z")
Hi We use a default filter for syslog messages, so they get parsed properly. However I wondered if it would be possible to match the message against a secondary pattern and extract new fields from the message: Dec 4 14… --- ## [Logstash not executing the configs](https://discuss.elastic.co/t/logstash-not-executing-the-configs/80201)
**Author:** [@gmmurugan](https://discuss.elastic.co/u/gmmurugan)\ **Replies:** 15\ **Last updated:** [March 29, 2017, 5:33am UTC](https://discuss.elastic.co/t/logstash-not-executing-the-configs/80201 "2017-03-29T05:33:35Z")
I upgraded to logstash 5 from 2.4 and resolved the logstash.yml and the path for that file also. whatever i do, there is no result. just hangs at the statement even i run manually. what am i missing. bin/logstash --pat… --- ## [Xml parsing issue with xpath](https://discuss.elastic.co/t/xml-parsing-issue-with-xpath/79834)
**Author:** [@kamran.matloob](https://discuss.elastic.co/u/kamran.matloob)\ **Replies:** 6\ **Last updated:** [March 28, 2017, 8:49pm UTC](https://discuss.elastic.co/t/xml-parsing-issue-with-xpath/79834 "2017-03-28T20:49:31Z")
hi here ! I am new to use logstash groking sepcially with xml. My task is parsing xml.I successfully wrote the GROK xml filter for a specific data XML format but then i came to know that data (XML) has different form… --- ## [V5.3 won't start on CentOS 7.3](https://discuss.elastic.co/t/v5-3-wont-start-on-centos-7-3/80348)
**Author:** [@gtirloni](https://discuss.elastic.co/u/gtirloni)\ **Replies:** 6\ **Last updated:** [March 28, 2017, 8:07pm UTC](https://discuss.elastic.co/t/v5-3-wont-start-on-centos-7-3/80348 "2017-03-28T20:07:22Z")
Installed logstash 5.3 from the official repository and tried to run the following command (from https://www.elastic.co/guide/en/logstash/current/first-event.html): # /usr/share/logstash/bin/logstash -e 'input { stdin … --- ## [Parse Exim logs](https://discuss.elastic.co/t/parse-exim-logs/80358)
**Author:** [@maxw2016](https://discuss.elastic.co/u/maxw2016)\ **Replies:** 0\ **Last updated:** [March 28, 2017, 7:43pm UTC](https://discuss.elastic.co/t/parse-exim-logs/80358 "2017-03-28T19:43:48Z")
Hello, I'm trying to parse Exim logs which spread through multiple lines Here is example of logs: 2017-03-28 12:26:49 1cswlV-0004KG-2c DKIM: d=gmail.com s=20161025 c=relaxed/relaxed a=rsa-sha256 b=2048 \[verification … --- ## [Cloudwatch logs input configuration help needed](https://discuss.elastic.co/t/cloudwatch-logs-input-configuration-help-needed/80359)
**Author:** [@tarp](https://discuss.elastic.co/u/tarp)\ **Replies:** 0\ **Last updated:** [March 28, 2017, 7:45pm UTC](https://discuss.elastic.co/t/cloudwatch-logs-input-configuration-help-needed/80359 "2017-03-28T19:45:01Z")
Hi, I'm trying to configure the cloudwatch input plugin to pull in custom cloudwatch logs. Below is my config. cloudwatch { namespace =\> "AWS/LOGS" metrics =\> \["IncomingBytes","IncomingLogEvents"\] filters … --- ## [Lumberjack is not able to open port](https://discuss.elastic.co/t/lumberjack-is-not-able-to-open-port/39275)
**Author:** [@NiteshBarnwal](https://discuss.elastic.co/u/NiteshBarnwal)\ **Replies:** 2\ **Last updated:** [March 28, 2017, 6:13pm UTC](https://discuss.elastic.co/t/lumberjack-is-not-able-to-open-port/39275 "2017-03-28T18:13:17Z")
Logstash is not getting started, though it was working few hours ago. seems like lumberjack is not able to open the port log trace: Starting lumberjack input listener {:address=\>"0.0.0.0:5534", :level=\>:info, :file=\>"… --- ## [Plugin Logstash-codec-nmap updated 3/17/17](https://discuss.elastic.co/t/plugin-logstash-codec-nmap-updated-3-17-17/79082)
**Author:** [@Dark\_Light](https://discuss.elastic.co/u/Dark_Light)\ **Replies:** 4\ **Last updated:** [March 28, 2017, 4:24pm UTC](https://discuss.elastic.co/t/plugin-logstash-codec-nmap-updated-3-17-17/79082 "2017-03-28T16:24:26Z")
The Logstash-codec-nmap plugin was updated and merged with master this morning. It does not appear that when installing using the logstash plugin manager that the update is available to logstash as the install still fa… --- ## [HELP for debug log stash config](https://discuss.elastic.co/t/help-for-debug-log-stash-config/80232)
**Author:** [@jroberto](https://discuss.elastic.co/u/jroberto)\ **Replies:** 2\ **Last updated:** [March 28, 2017, 3:35pm UTC](https://discuss.elastic.co/t/help-for-debug-log-stash-config/80232 "2017-03-28T15:35:52Z")
Hello guys! I want to evalute a custom log. When I test %{HSMFORMAT} on the online grok debugger has no problem with that. If i use it in my config file i got an \_grokparsefailure tag. My logstash version is 5.2.2 T… --- ## [Index logstash in kibana](https://discuss.elastic.co/t/index-logstash-in-kibana/80286)
**Author:** [@Sujith](https://discuss.elastic.co/u/Sujith)\ **Replies:** 4\ **Last updated:** [March 28, 2017, 3:11pm UTC](https://discuss.elastic.co/t/index-logstash-in-kibana/80286 "2017-03-28T15:11:35Z")
How to index logstash-\* in kibana. Im using ELK stack 5.2 version and i have indexed filebeat, metricbeat, packetbeat and winlogbeat. Im using filbeat config sending logs to logstash-\>elasticsearch-\>kibana i need t… --- ## [Logstash metric filter - help](https://discuss.elastic.co/t/logstash-metric-filter-help/80313)
**Author:** [@August\_rush](https://discuss.elastic.co/u/August_rush)\ **Replies:** 2\ **Last updated:** [March 28, 2017, 3:09pm UTC](https://discuss.elastic.co/t/logstash-metric-filter-help/80313 "2017-03-28T15:09:16Z")
The example log file contains a list of logs , i am trying to apply a metric filter to check the count of log-level==ERROR for every 5 s and if the count is greater than 1 then it should trigger an email, i am not sur… --- ## [Need to populate geo location based on US State name column using logstash,es and kibi](https://discuss.elastic.co/t/need-to-populate-geo-location-based-on-us-state-name-column-using-logstash-es-and-kibi/78755)
**Author:** [@navtej.billing](https://discuss.elastic.co/u/navtej.billing)\ **Replies:** 10\ **Last updated:** [March 28, 2017, 2:45pm UTC](https://discuss.elastic.co/t/need-to-populate-geo-location-based-on-us-state-name-column-using-logstash-es-and-kibi/78755 "2017-03-28T14:45:35Z")
I have a field with the name of USA states, I need to add another field which should have the geo location for that state. So, that I can use the Tile map in Kibi based on these newly generated Geo location. My Logstas… --- ## [Unable to install custom plugin in logstash5.x](https://discuss.elastic.co/t/unable-to-install-custom-plugin-in-logstash5-x/78141)
**Author:** [@Nikhil\_Pillai](https://discuss.elastic.co/u/Nikhil_Pillai)\ **Replies:** 1\ **Last updated:** [March 28, 2017, 2:20pm UTC](https://discuss.elastic.co/t/unable-to-install-custom-plugin-in-logstash5-x/78141 "2017-03-28T14:20:15Z")
Hi, I am facing below error when trying to install a custom (logstash-output-jdbc) plugin in logstash 5.2. Server doesn't have an internet access. ./logstash-plugin install file:///home/tibusr/mounted/ELK/logstash-o… --- ## [Problems starting logstash](https://discuss.elastic.co/t/problems-starting-logstash/80279)
**Author:** [@Meikop](https://discuss.elastic.co/u/Meikop)\ **Replies:** 2\ **Last updated:** [March 28, 2017, 1:50pm UTC](https://discuss.elastic.co/t/problems-starting-logstash/80279 "2017-03-28T13:50:48Z")
Hello, I'm exploring logstash to solve some of our logging-related problems but ran into a weird issue: 1) I installed logstash onto Ubuntu 16.04.2 LTS via APT (https://www.elastic.co/guide/en/logstash/2.3/installing… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1008) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1010)