# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1142 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1143 --- ## [Ingesting meta data from database and file from file system](https://discuss.elastic.co/t/ingesting-meta-data-from-database-and-file-from-file-system/52651)
**Author:** [@muddana](https://discuss.elastic.co/u/muddana)\ **Replies:** 1\ **Last updated:** [June 14, 2016, 6:37am UTC](https://discuss.elastic.co/t/ingesting-meta-data-from-database-and-file-from-file-system/52651 "2016-06-14T06:37:35Z")
Hi - New to logstash. I am looking to ingest files along with related metadata into ES. I have metadata and file path in database. Just wondering if there is a way to get meta data (like created date), and file … --- ## [Logstash service start but later stopped with error he error reported is: \\n java.io.EOFException"](https://discuss.elastic.co/t/logstash-service-start-but-later-stopped-with-error-he-error-reported-is-n-java-io-eofexception/52639)
**Author:** [@Li\_Zhang](https://discuss.elastic.co/u/Li_Zhang)\ **Replies:** 1\ **Last updated:** [June 14, 2016, 6:35am UTC](https://discuss.elastic.co/t/logstash-service-start-but-later-stopped-with-error-he-error-reported-is-n-java-io-eofexception/52639 "2016-06-14T06:35:04Z")
Can someone give some suggestions about following issue? logstash1.5-1.5.2-4.el6. Not Running Error Seen : {:timestamp=\>"2016-06-10T05:28:24.691000+0000", :message=\>"weblogic plugin is using the 'milestone' method to… --- ## [Logstash won't read 1 line input file (and might miss last line of data in any file)](https://discuss.elastic.co/t/logstash-wont-read-1-line-input-file-and-might-miss-last-line-of-data-in-any-file/51855)
**Author:** [@itaydvir](https://discuss.elastic.co/u/itaydvir)\ **Replies:** 4\ **Last updated:** [June 14, 2016, 6:27am UTC](https://discuss.elastic.co/t/logstash-wont-read-1-line-input-file-and-might-miss-last-line-of-data-in-any-file/51855 "2016-06-14T06:27:15Z")
I spent a lot of time on this issue until I found the problem: Like many people who start using logstash, I have created 1 demo input file with 1 line of data. When I used stdin() for the same input line, everything work… --- ## [Does logstash-jmx plugin support wildcards](https://discuss.elastic.co/t/does-logstash-jmx-plugin-support-wildcards/42489)
**Author:** [@Sai\_Birada](https://discuss.elastic.co/u/Sai_Birada)\ **Replies:** 2\ **Last updated:** [June 14, 2016, 5:50am UTC](https://discuss.elastic.co/t/does-logstash-jmx-plugin-support-wildcards/42489 "2016-06-14T05:50:53Z")
I am trying to monitor my kafka cluster, I am pulling metrics through logstash-jmx plugin and pushing them to elasticsearch for monitoring. Kafka is providing per topic level metrics which i want to visualize. So is ther… --- ## [Unable to view catalina.out file in Kibana](https://discuss.elastic.co/t/unable-to-view-catalina-out-file-in-kibana/52344)
**Author:** [@rvaedex23](https://discuss.elastic.co/u/rvaedex23)\ **Replies:** 3\ **Last updated:** [June 14, 2016, 5:37am UTC](https://discuss.elastic.co/t/unable-to-view-catalina-out-file-in-kibana/52344 "2016-06-14T05:37:16Z")
I have moved some catalina.out logs from a Production server to the test environment this is where elasticsearch is installed. I am not able to view the logs. I have set up input/filter/output and I am not seeing the … --- ## [Can Logstash accepts JMX object name of wildcards?](https://discuss.elastic.co/t/can-logstash-accepts-jmx-object-name-of-wildcards/52680)
**Author:** [@natarajan\_k](https://discuss.elastic.co/u/natarajan_k)\ **Replies:** 0\ **Last updated:** [June 14, 2016, 2:07am UTC](https://discuss.elastic.co/t/can-logstash-accepts-jmx-object-name-of-wildcards/52680 "2016-06-14T02:07:36Z")
i am collecting the metrics from kafka through logstash and storing it in elastic search. basically, i can able to collect the server metrics from kafka but metrics of producer and consumer is not working. below are … --- ## [Logstash doesn't capture Retweet?](https://discuss.elastic.co/t/logstash-doesnt-capture-retweet/52665)
**Author:** [@Gautam\_S\_Thakur](https://discuss.elastic.co/u/Gautam_S_Thakur)\ **Replies:** 1\ **Last updated:** [June 13, 2016, 11:37pm UTC](https://discuss.elastic.co/t/logstash-doesnt-capture-retweet/52665 "2016-06-13T23:37:43Z")
I am trying to use Logstash to collect Twitter Steaming data. Every thing looks great, but one thing is missing I guess. It seems that Logstash is not injecting Retweets in ES. Any idea? btw, ignore\_retweets is set to fa… --- ## [Grok Filter Pattern with Comma](https://discuss.elastic.co/t/grok-filter-pattern-with-comma/52642)
**Author:** [@sarthak2016](https://discuss.elastic.co/u/sarthak2016)\ **Replies:** 7\ **Last updated:** [June 13, 2016, 8:13pm UTC](https://discuss.elastic.co/t/grok-filter-pattern-with-comma/52642 "2016-06-13T20:13:12Z")
Hello All, I need %{GREEDYDATA:Text1} till first comma Here is the input file data 02/06/2016 01:01:07,Text1Data,Text2Datap1,Text2Datap2,Text2Datap3,1046,547,error Grok Expression %{DATESTAMP:Time},%{GREEDYDATA:Tex… --- ## [Logstash crashes then brings Kibana down with it](https://discuss.elastic.co/t/logstash-crashes-then-brings-kibana-down-with-it/52354)
**Author:** [@jiayo](https://discuss.elastic.co/u/jiayo)\ **Replies:** 1\ **Last updated:** [June 13, 2016, 8:09pm UTC](https://discuss.elastic.co/t/logstash-crashes-then-brings-kibana-down-with-it/52354 "2016-06-13T20:09:32Z")
Hello elastic community, I get this error intermittently (now constantly) Logstash is located to the left hand side of the screen. This error takes up the whole screen and I am not able to go find the beginning of it… --- ## [Filebeat -\> logstash ordering](https://discuss.elastic.co/t/filebeat-logstash-ordering/52591)
**Author:** [@Donovan](https://discuss.elastic.co/u/Donovan)\ **Replies:** 3\ **Last updated:** [June 13, 2016, 4:57pm UTC](https://discuss.elastic.co/t/filebeat-logstash-ordering/52591 "2016-06-13T16:57:10Z")
i have a stream. filebeat-\>logstash-\>(s3, kafka) sometimes, it does not guarantee ordering. what is the checkpoint to solve this problem? # filebeat conf input { beats { port =\> 5100 type =\> "rhym… --- ## [Logstash eventlog filter](https://discuss.elastic.co/t/logstash-eventlog-filter/51316)
**Author:** [@babeesh](https://discuss.elastic.co/u/babeesh)\ **Replies:** 25\ **Last updated:** [June 13, 2016, 12:47pm UTC](https://discuss.elastic.co/t/logstash-eventlog-filter/51316 "2016-06-13T12:47:34Z")
I need to filter error messages from eventlog from logstash. logstash configuration input { tcp { port =\> 5045 type =\> 'eventlog' } } filter{ if \[type\] == 'eventlog' { if \[Severity\] == "ERROR" {… --- ## [Checking for a pattern in a logstash field name](https://discuss.elastic.co/t/checking-for-a-pattern-in-a-logstash-field-name/52601)
**Author:** [@quietrags](https://discuss.elastic.co/u/quietrags)\ **Replies:** 1\ **Last updated:** [June 13, 2016, 12:02pm UTC](https://discuss.elastic.co/t/checking-for-a-pattern-in-a-logstash-field-name/52601 "2016-06-13T12:02:06Z")
I would like to check if a dynamic field has a certain pattern in it - for example in my case i get a lot of '.' in the field name which elastic search complains about whilst indexing. Is there a way i can check if th… --- ## [Difference between filebeat parsing & stdin parsing of the same log format](https://discuss.elastic.co/t/difference-between-filebeat-parsing-stdin-parsing-of-the-same-log-format/52493)
**Author:** [@MatthewDaniels](https://discuss.elastic.co/u/MatthewDaniels)\ **Replies:** 6\ **Last updated:** [June 13, 2016, 11:59am UTC](https://discuss.elastic.co/t/difference-between-filebeat-parsing-stdin-parsing-of-the-same-log-format/52493 "2016-06-13T11:59:30Z")
Hi there, I am having some difficultiy with a custom log format, when sending it into logstash via the beats input. If I send the same log line into logstash (with the same filters as the beats input) via stdin, the ev… --- ## [Cisco ASA Patterns in Logstash 1.5.1](https://discuss.elastic.co/t/cisco-asa-patterns-in-logstash-1-5-1/24661)
**Author:** [@DavidSims](https://discuss.elastic.co/u/DavidSims)\ **Replies:** 9\ **Last updated:** [June 13, 2016, 11:25am UTC](https://discuss.elastic.co/t/cisco-asa-patterns-in-logstash-1-5-1/24661 "2016-06-13T11:25:22Z")
I have yet to get logstash to parse Cisco ASA logs correctly. I have tried every tutorial a could find and I ALWAYS get \_grokparsefailure. I stripped it all the way down and it seems to hate the patterns. filter { g… --- ## [RESOLVED Logstash fails when output set to elasticsearch "SSLConnectionSocketFactory not found in packages"](https://discuss.elastic.co/t/resolved-logstash-fails-when-output-set-to-elasticsearch-sslconnectionsocketfactory-not-found-in-packages/41407)
**Author:** [@dmx555](https://discuss.elastic.co/u/dmx555)\ **Replies:** 2\ **Last updated:** [June 13, 2016, 10:43am UTC](https://discuss.elastic.co/t/resolved-logstash-fails-when-output-set-to-elasticsearch-sslconnectionsocketfactory-not-found-in-packages/41407 "2016-06-13T10:43:39Z")
Someone else is probably going to experience the same problem so thought I'd explain how I solve it : simply unset CLASSPATH. AS I am running from a shell which runs HDFS commands and this already sets up a CLASSPATH. l… --- ## [Many Logstash to get HA](https://discuss.elastic.co/t/many-logstash-to-get-ha/52592)
**Author:** [@Guillermo\_Ortiz](https://discuss.elastic.co/u/Guillermo_Ortiz)\ **Replies:** 0\ **Last updated:** [June 13, 2016, 10:23am UTC](https://discuss.elastic.co/t/many-logstash-to-get-ha/52592 "2016-06-13T10:23:11Z")
I want to have HA in Logstash, my scenario is: N-Flume --\> Kafka --\> N Logstash --\> ElasticSearch I don't know if there are any recommendation about HA, I read https://www.elastic.co/guide/en/logstash/current/deploying… --- ## [Json filter and removing json fields](https://discuss.elastic.co/t/json-filter-and-removing-json-fields/52556)
**Author:** [@quietrags](https://discuss.elastic.co/u/quietrags)\ **Replies:** 1\ **Last updated:** [June 13, 2016, 8:43am UTC](https://discuss.elastic.co/t/json-filter-and-removing-json-fields/52556 "2016-06-13T08:43:42Z")
I am using the json filter in my logstash processing and i would like to remove some of the json fields that get parsed. I am unable to figure out what the format should be. Here is my json { "cookies" : {"x" : 1, "mo… --- ## [Merge Several json parsed fields to 1 field](https://discuss.elastic.co/t/merge-several-json-parsed-fields-to-1-field/52065)
**Author:** [@Wesley\_Smekens](https://discuss.elastic.co/u/Wesley_Smekens)\ **Replies:** 4\ **Last updated:** [June 13, 2016, 8:18am UTC](https://discuss.elastic.co/t/merge-several-json-parsed-fields-to-1-field/52065 "2016-06-13T08:18:41Z")
Hey All, Part of our logs we recieve are in json format. I filter out these messages with following filter: json { source =\> "Request" target =\> "Request\_parsed" } This works perfectly and we are getting a… --- ## [Multiline allow\_duplicates=\>true config eleminating duplicates](https://discuss.elastic.co/t/multiline-allow-duplicates-true-config-eleminating-duplicates/51304)
**Author:** [@Avishek](https://discuss.elastic.co/u/Avishek)\ **Replies:** 3\ **Last updated:** [June 13, 2016, 8:08am UTC](https://discuss.elastic.co/t/multiline-allow-duplicates-true-config-eleminating-duplicates/51304 "2016-06-13T08:08:30Z")
Hello, I am using logstash 2.3.1. I have created a config where I have used a multiline filter as one event in my log spans accross multiple lines. Apart from other filters which I have in my config, my last fi… --- ## [How to get the date object substring in logstash filter](https://discuss.elastic.co/t/how-to-get-the-date-object-substring-in-logstash-filter/49760)
**Author:** [@baneshbanna](https://discuss.elastic.co/u/baneshbanna)\ **Replies:** 13\ **Last updated:** [June 13, 2016, 7:31am UTC](https://discuss.elastic.co/t/how-to-get-the-date-object-substring-in-logstash-filter/49760 "2016-06-13T07:31:53Z")
In Logstash filter i want to use yyyy-mm . I should get yyyy-mm from date object using substring function. How to get the substring of date object in logstash filter Thanks in advance. --- ## [lOGSTASH cONFIGURATION not working in 2.3.3](https://discuss.elastic.co/t/logstash-configuration-not-working-in-2-3-3/52522)
**Author:** [@Hassan\_Nashrah](https://discuss.elastic.co/u/Hassan_Nashrah)\ **Replies:** 1\ **Last updated:** [June 13, 2016, 1:41am UTC](https://discuss.elastic.co/t/logstash-configuration-not-working-in-2-3-3/52522 "2016-06-13T01:41:10Z")
My logstash configuration is not working in the latest update of logstash it works with logstash 1.4.4 ... The given configuration is invalid. Reason: The setting \`type\` in plugin \`grok\` is obsolete and is no longer a… --- ## [Logstash, collectd and avoiding duplicated data in the ES](https://discuss.elastic.co/t/logstash-collectd-and-avoiding-duplicated-data-in-the-es/52548)
**Author:** [@jasonh](https://discuss.elastic.co/u/jasonh)\ **Replies:** 0\ **Last updated:** [June 12, 2016, 10:01pm UTC](https://discuss.elastic.co/t/logstash-collectd-and-avoiding-duplicated-data-in-the-es/52548 "2016-06-12T22:01:39Z")
Hi Architectural question here I'm setting up an ELK environment There are 2 logstash servers feeding into the ES cluster Each of the Logstash will have a collectd instance plugin snmp polling the same devices for data r… --- ## [Grok Patterns? for OpenVPN](https://discuss.elastic.co/t/grok-patterns-for-openvpn/52265)
**Author:** [@killmasta93](https://discuss.elastic.co/u/killmasta93)\ **Replies:** 2\ **Last updated:** [June 12, 2016, 7:45pm UTC](https://discuss.elastic.co/t/grok-patterns-for-openvpn/52265 "2016-06-12T19:45:02Z")
Hi, I Have been trying for many months and no luck on getting this to work to filter the logs from pfSense and Send them to ELK. I am not sure where i should be looking at? This is what I have filter{ if \[type\] == "O… --- ## [Json target based field value (or tag)?](https://discuss.elastic.co/t/json-target-based-field-value-or-tag/49350)
**Author:** [@robinpblanchard](https://discuss.elastic.co/u/robinpblanchard)\ **Replies:** 3\ **Last updated:** [June 12, 2016, 2:05pm UTC](https://discuss.elastic.co/t/json-target-based-field-value-or-tag/49350 "2016-06-12T14:05:16Z")
Hi, I'd like to dynamically set the json target based on a field value. I tried: target =\> "%{field}" with no success. Is there an easy way to accomplish this? --- ## [Add Logstash to autorun services](https://discuss.elastic.co/t/add-logstash-to-autorun-services/52296)
**Author:** [@bibl2008](https://discuss.elastic.co/u/bibl2008)\ **Replies:** 1\ **Last updated:** [June 12, 2016, 11:21am UTC](https://discuss.elastic.co/t/add-logstash-to-autorun-services/52296 "2016-06-12T11:21:34Z")
Hello, Please help. 1. I download Logstash 2.3.1 All Plugins 2. unzip to /opt/elastic/logstash-2.3.1/ 3. in /opt/elastic/logstash-2.3.1/bin add config.conf: input { beats { port =\> 50… --- ## [Read the directory file name with logstash](https://discuss.elastic.co/t/read-the-directory-file-name-with-logstash/52283)
**Author:** [@Kennedy\_Kan1](https://discuss.elastic.co/u/Kennedy_Kan1)\ **Replies:** 1\ **Last updated:** [June 12, 2016, 11:19am UTC](https://discuss.elastic.co/t/read-the-directory-file-name-with-logstash/52283 "2016-06-12T11:19:28Z")
I have logs file located in C:/\/log/\/\/access.\*.log I have written a logstash conf file to read the logs but how can I also read the directory name wlsDomain in logstash as a new field… --- ## [Parsing repeated patterns in logstash](https://discuss.elastic.co/t/parsing-repeated-patterns-in-logstash/51882)
**Author:** [@Deepak\_Patnaik](https://discuss.elastic.co/u/Deepak_Patnaik)\ **Replies:** 6\ **Last updated:** [June 11, 2016, 1:55pm UTC](https://discuss.elastic.co/t/parsing-repeated-patterns-in-logstash/51882 "2016-06-11T13:55:16Z")
Continuing the discussion from Parsing repeated patterns: In my log file i have the following pattern: 2014-02-19 19:13:12|oAmwic|20140219190348\_rmzXR|20140219190348\_rmzXR|ADR\_TST|Dim\_State\_Trade|Default|6|Java Excepti… --- ## [Logstash GeoIP filter (Tile map)](https://discuss.elastic.co/t/logstash-geoip-filter-tile-map/52463)
**Author:** [@bishaka](https://discuss.elastic.co/u/bishaka)\ **Replies:** 1\ **Last updated:** [June 11, 2016, 6:41am UTC](https://discuss.elastic.co/t/logstash-geoip-filter-tile-map/52463 "2016-06-11T06:41:09Z")
Hi, I am trying to use GeoIP filter to display geographical location of IP addresses. I followed this tutorial https://www.digitalocean.com/community/tutorials/how-to-map-user-location-with-geoip-and-elk-elasticsearch-lo… --- ## [Arithmetic with string](https://discuss.elastic.co/t/arithmetic-with-string/52131)
**Author:** [@stecino](https://discuss.elastic.co/u/stecino)\ **Replies:** 5\ **Last updated:** [June 10, 2016, 7:44pm UTC](https://discuss.elastic.co/t/arithmetic-with-string/52131 "2016-06-10T19:44:09Z")
Hello, I have a string "5.34" and I am trying to multiply it by number, say 1000. Can I do it with gsub or ruby plugin. I have read that ruby plugin is resource intensive, so i would prefer to do it with gsub if possib… --- ## [JDBC connect String Password Encryption](https://discuss.elastic.co/t/jdbc-connect-string-password-encryption/47892)
**Author:** [@daivenu](https://discuss.elastic.co/u/daivenu)\ **Replies:** 1\ **Last updated:** [June 10, 2016, 7:19pm UTC](https://discuss.elastic.co/t/jdbc-connect-string-password-encryption/47892 "2016-06-10T19:19:35Z")
i would like to know how to use encryp password in JDBC jdbcplugin.conf file i need to encrypt the below jdbc\_password, since i cant use as clear text. please let me know if any other info required input { jdbc { jdb… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1141) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1143)