# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=959 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 960 --- ## [Read different lines of single log file having different pattern](https://discuss.elastic.co/t/read-different-lines-of-single-log-file-having-different-pattern/89536)
**Author:** [@richagautam](https://discuss.elastic.co/u/richagautam)\ **Replies:** 2\ **Last updated:** [June 23, 2017, 9:28am UTC](https://discuss.elastic.co/t/read-different-lines-of-single-log-file-having-different-pattern/89536 "2017-06-23T09:28:54Z")
Hi , 2/17/2017 16:11:14 hostname some text some text 2/17/2017 16:11:14 hostname Filename !!Begin test: Scriptname: abcTest/output/outputfile.log 2/17/2017 16:11:14 hostname Filename test data test data test data test … --- ## [Read Particular line from log File](https://discuss.elastic.co/t/read-particular-line-from-log-file/90256)
**Author:** [@richagautam](https://discuss.elastic.co/u/richagautam)\ **Replies:** 1\ **Last updated:** [June 23, 2017, 9:27am UTC](https://discuss.elastic.co/t/read-particular-line-from-log-file/90256 "2017-06-23T09:27:04Z")
Hi , I have to read 3 different lines from log files based on some text and then output the fields in a csv file. sample log data:- 20110607 095826 \[.\] !! Begin test. Script filename/text.txt 20110607 095826 \[.\] … --- ## [Logstash avro file problem](https://discuss.elastic.co/t/logstash-avro-file-problem/90601)
**Author:** [@NandhaKrishna](https://discuss.elastic.co/u/NandhaKrishna)\ **Replies:** 0\ **Last updated:** [June 23, 2017, 9:05am UTC](https://discuss.elastic.co/t/logstash-avro-file-problem/90601 "2017-06-23T09:05:01Z")
I'm trying to input avro file and link it to its avsc file and write that into elasticsearch. This is my config file, input { file { path =\> "twitter.avro" type =\> "avro" codec =\> { avro =\> { schema\_uri =\> "twi… --- ## [When logstash is stopped it goes to failed state](https://discuss.elastic.co/t/when-logstash-is-stopped-it-goes-to-failed-state/90580)
**Author:** [@DharaniKumar](https://discuss.elastic.co/u/DharaniKumar)\ **Replies:** 0\ **Last updated:** [June 23, 2017, 6:37am UTC](https://discuss.elastic.co/t/when-logstash-is-stopped-it-goes-to-failed-state/90580 "2017-06-23T06:37:22Z")
This is what im getting when i stop logstash...Some one help me get over this. root@dev01:~# service logstash stop root@dev01:~# service logstash status ● logstash.service - logstash Loaded: loaded (/etc/systemd/sys… --- ## [Logstash drop filter is not dropping events](https://discuss.elastic.co/t/logstash-drop-filter-is-not-dropping-events/27396)
**Author:** [@Rubaiyat\_Islam\_Sadat](https://discuss.elastic.co/u/Rubaiyat_Islam_Sadat)\ **Replies:** 1\ **Last updated:** [June 23, 2017, 6:14am UTC](https://discuss.elastic.co/t/logstash-drop-filter-is-not-dropping-events/27396 "2017-06-23T06:14:11Z")
I am having a very strange situation where I have the following filter rule if \[logger\_name\] == "REPORT" { drop {} } and I expect that any events with my field\_name == "logger\_name" with that specific value will be d… --- ## [Split event by an object of objects into separate events](https://discuss.elastic.co/t/split-event-by-an-object-of-objects-into-separate-events/90382)
**Author:** [@berg](https://discuss.elastic.co/u/berg)\ **Replies:** 2\ **Last updated:** [June 22, 2017, 11:43pm UTC](https://discuss.elastic.co/t/split-event-by-an-object-of-objects-into-separate-events/90382 "2017-06-22T23:43:19Z")
I am attempting to use the logstash-filter-metric plugin to produce metrics split by the source \[host\] of the events. We have \> 1000 different log sources, and being able to track where all the logs are coming from acros… --- ## [How to remove IP-address from elasticsearch submission after geoip parses it?](https://discuss.elastic.co/t/how-to-remove-ip-address-from-elasticsearch-submission-after-geoip-parses-it/90504)
**Author:** [@UnitedMarsupials](https://discuss.elastic.co/u/UnitedMarsupials)\ **Replies:** 2\ **Last updated:** [June 22, 2017, 11:36pm UTC](https://discuss.elastic.co/t/how-to-remove-ip-address-from-elasticsearch-submission-after-geoip-parses-it/90504 "2017-06-22T23:36:09Z")
Our logs contain clients' IP-addresses, which we pass to the geoip-filter: filter { geoip { source =\> "client\_ip" } ... output { elasticsearch { hosts =\> \["...."\] index =\> "logstash-%{+YYYY.MM.dd}" } .… --- ## [Logstash-Slack plugin attachment format issue](https://discuss.elastic.co/t/logstash-slack-plugin-attachment-format-issue/90372)
**Author:** [@Bob04](https://discuss.elastic.co/u/Bob04)\ **Replies:** 1\ **Last updated:** [June 22, 2017, 8:40pm UTC](https://discuss.elastic.co/t/logstash-slack-plugin-attachment-format-issue/90372 "2017-06-22T20:40:34Z")
Does anybody know who to configure Logstash Slack output section in logstash.conf? I tried to do the following using this docs https://github.com/cyli/logstash-output-slack: slack { url =\> "..." channel =\> \["...… --- ## [Scaling Logstash](https://discuss.elastic.co/t/scaling-logstash/90158)
**Author:** [@Rory](https://discuss.elastic.co/u/Rory)\ **Replies:** 9\ **Last updated:** [June 22, 2017, 8:40pm UTC](https://discuss.elastic.co/t/scaling-logstash/90158 "2017-06-22T20:40:30Z")
Hi All, We have a system where we have to rebuild an index every night which has 300 Million documents. Our source of truth is SQL Server and every night we want to build a new index. Our document size is about 2-3 KB.… --- ## [External lookups in a logstash pipeline](https://discuss.elastic.co/t/external-lookups-in-a-logstash-pipeline/89758)
**Author:** [@Jaspreet\_Singh](https://discuss.elastic.co/u/Jaspreet_Singh)\ **Replies:** 7\ **Last updated:** [June 22, 2017, 6:49pm UTC](https://discuss.elastic.co/t/external-lookups-in-a-logstash-pipeline/89758 "2017-06-22T18:49:04Z")
So I have a requirement that goes something like this ... 1. An event arrives in kafka 2. Read that event from logstash and get an ID out of it 3. Look up that ID to fetch a JSON from say object storage 4. Do stuff l… --- ## [The definition of grok pattern CISCO\_REASON](https://discuss.elastic.co/t/the-definition-of-grok-pattern-cisco-reason/90373)
**Author:** [@Shuxin\_Lin](https://discuss.elastic.co/u/Shuxin_Lin)\ **Replies:** 2\ **Last updated:** [June 22, 2017, 5:49pm UTC](https://discuss.elastic.co/t/the-definition-of-grok-pattern-cisco-reason/90373 "2017-06-22T17:49:59Z")
My log entry is like: 2017-05-16T03:48:19.191162+00:00 ip-10-133-8-70 tcp\[18916\]: LOG Statistics \[Sender\] .... In this example, "LOG Statitics" can be extracted by pre-defined pattern: CISCO\_REASON But I cannot f… --- ## [My "Logstash.conf" gives errors](https://discuss.elastic.co/t/my-logstash-conf-gives-errors/89041)
**Author:** [@MartinNouv](https://discuss.elastic.co/u/MartinNouv)\ **Replies:** 31\ **Last updated:** [June 22, 2017, 5:39pm UTC](https://discuss.elastic.co/t/my-logstash-conf-gives-errors/89041 "2017-06-22T17:39:30Z")
Hello there ! I'm trying to configure an ELK to have a better overview of my CSV files. But when I launch my conf file, I get some issues I can't solve despite my searches and my tries for solving. My conf file : … --- ## [Field named "tags" in logstash event causing problem](https://discuss.elastic.co/t/field-named-tags-in-logstash-event-causing-problem/89987)
**Author:** [@Jaspreet\_Singh](https://discuss.elastic.co/u/Jaspreet_Singh)\ **Replies:** 1\ **Last updated:** [June 22, 2017, 5:30pm UTC](https://discuss.elastic.co/t/field-named-tags-in-logstash-event-causing-problem/89987 "2017-06-22T17:30:07Z")
Hi, Here is what my configuration looks like ... filter { mutate { rename =\> { "keywords" =\> "\[tags\]\[keywords\]" } } } Essentially, I'm looking to mutate into a nested JSON under field n… --- ## [How to call web service with JSON parameters in logstash/Filebeat](https://discuss.elastic.co/t/how-to-call-web-service-with-json-parameters-in-logstash-filebeat/90389)
**Author:** [@patil-akshay](https://discuss.elastic.co/u/patil-akshay)\ **Replies:** 1\ **Last updated:** [June 22, 2017, 3:16pm UTC](https://discuss.elastic.co/t/how-to-call-web-service-with-json-parameters-in-logstash-filebeat/90389 "2017-06-22T15:16:19Z")
I have an web service which takes parameters as {"From\_TimeStamp":"2016-09-30 13:47:13","To\_TimeStamp":"2016-10-05 13:47:13"} which returns json output . How can I achive the same --- ## [Convert uppercase in lowercase](https://discuss.elastic.co/t/convert-uppercase-in-lowercase/90416)
**Author:** [@erion](https://discuss.elastic.co/u/erion)\ **Replies:** 12\ **Last updated:** [June 22, 2017, 1:44pm UTC](https://discuss.elastic.co/t/convert-uppercase-in-lowercase/90416 "2017-06-22T13:44:59Z")
Hi, i've tried to convert field.keyword in lowrcase by using this mapping: filter { mutate { lowercase =\> { match =\> \[ "request"\] } } grok { match =\> { "message" =\> '%{IPORHOST:clien… --- ## [Csv eof](https://discuss.elastic.co/t/csv-eof/90405)
**Author:** [@Mario\_van\_Gemert](https://discuss.elastic.co/u/Mario_van_Gemert)\ **Replies:** 6\ **Last updated:** [June 22, 2017, 1:10pm UTC](https://discuss.elastic.co/t/csv-eof/90405 "2017-06-22T13:10:08Z")
I have a CSV file which I imported with Logstash into Elasticsearch. This CSV is "growing" when time goes by. But when this happens Logstash completely imports the file again, this in stead of the last added records. I a… --- ## [Logstash if and Mutate hostname](https://discuss.elastic.co/t/logstash-if-and-mutate-hostname/90449)
**Author:** [@Sinethra\_Seneviratne](https://discuss.elastic.co/u/Sinethra_Seneviratne)\ **Replies:** 1\ **Last updated:** [June 22, 2017, 12:40pm UTC](https://discuss.elastic.co/t/logstash-if-and-mutate-hostname/90449 "2017-06-22T12:40:55Z")
hi, im trying to filter my S3 logs and change Original value of the " cs\_host " field to a different one, This is the filter block of my Logstash conf. But this isn't working. Please let me know whats wrong here. filte… --- ## [Parsing multiline logs in Logstash](https://discuss.elastic.co/t/parsing-multiline-logs-in-logstash/89676)
**Author:** [@nmane](https://discuss.elastic.co/u/nmane)\ **Replies:** 3\ **Last updated:** [June 22, 2017, 12:22pm UTC](https://discuss.elastic.co/t/parsing-multiline-logs-in-logstash/89676 "2017-06-22T12:22:31Z")
I am using Logstash to parse my multiline logs. I have used multiline in filebeat. I am able to use grok filter for the first line of the log. But the pattern is not same for all multiline logs. For example, my logs are: … --- ## [Logstash configtest error..logstash 5.4](https://discuss.elastic.co/t/logstash-configtest-error-logstash-5-4/90437)
**Author:** [@Jonesthomas](https://discuss.elastic.co/u/Jonesthomas)\ **Replies:** 23\ **Last updated:** [June 22, 2017, 11:37am UTC](https://discuss.elastic.co/t/logstash-configtest-error-logstash-5-4/90437 "2017-06-22T11:37:47Z")
root@dev01:/usr/share/logstash/bin# ./logstash -configtest -e WARNING: Could not find logstash.yml which is typically located in $LS\_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuin… --- ## [Calculate elapsed time without "start" or "end" message](https://discuss.elastic.co/t/calculate-elapsed-time-without-start-or-end-message/89734)
**Author:** [@abrejuin](https://discuss.elastic.co/u/abrejuin)\ **Replies:** 4\ **Last updated:** [June 22, 2017, 10:21am UTC](https://discuss.elastic.co/t/calculate-elapsed-time-without-start-or-end-message/89734 "2017-06-22T10:21:02Z")
Hello, I have some logs generated by a jobs. Each job occur have a unique id but there is no specific start message and no specific end message. I am wondering how can I measured elapsed time between first and last l… --- ## [Worker thread consumes 100 % CPU (for no reason)](https://discuss.elastic.co/t/worker-thread-consumes-100-cpu-for-no-reason/261)
**Author:** [@kaaelhaa](https://discuss.elastic.co/u/kaaelhaa)\ **Replies:** 4\ **Last updated:** [May 8, 2015, 7:10pm UTC](https://discuss.elastic.co/t/worker-thread-consumes-100-cpu-for-no-reason/261 "2015-05-08T19:10:15Z")
Hi, I've recently set up a full ELK-stack. Things are not playing nice though. I'm experiencing the Logstash worker thread is constantly consuming 100 % of one of my CPU cores, even when there are nothing to ship or pa… --- ## [Unable to install the logstash plugin logstash-filter-fix\_protocol](https://discuss.elastic.co/t/unable-to-install-the-logstash-plugin-logstash-filter-fix-protocol/88599)
**Author:** [@mussa572](https://discuss.elastic.co/u/mussa572)\ **Replies:** 4\ **Last updated:** [June 22, 2017, 9:52am UTC](https://discuss.elastic.co/t/unable-to-install-the-logstash-plugin-logstash-filter-fix-protocol/88599 "2017-06-22T09:52:13Z")
Hello All when I try to install the Logstsh plugin downloaded directly from Github (https://github.com/connamara/logstash-filter-fix\_protocol) I get the following message . bin/logstash-plugin install file:///usr/sh… --- ## [Parsine multiline in logstash](https://discuss.elastic.co/t/parsine-multiline-in-logstash/89919)
**Author:** [@baha1](https://discuss.elastic.co/u/baha1)\ **Replies:** 2\ **Last updated:** [June 22, 2017, 9:43am UTC](https://discuss.elastic.co/t/parsine-multiline-in-logstash/89919 "2017-06-22T09:43:24Z")
Hi every body, i have a log file like below: Line1 : 17:13:41,971 ERROR \[org.apache.catalina.core.ContainerBase.\[jboss.web\].\[default-host\]\] (http-/0.0.0.0:10805-7) JBWEB000313: Exception processing error page /error/er… --- ## [Date FIlter - \_dateparsefailure \[SOLVED\]](https://discuss.elastic.co/t/date-filter--dateparsefailure-solved/64692)
**Author:** [@Clement\_Ros](https://discuss.elastic.co/u/Clement_Ros)\ **Replies:** 20\ **Last updated:** [June 22, 2017, 8:28am UTC](https://discuss.elastic.co/t/date-filter--dateparsefailure-solved/64692 "2017-06-22T08:28:53Z")
Hi, I'm new with the date filter. I set up this configuration : filter { if "mail" in \[type\] and "maillog" in \[source\] { grok { match =\> \["message","%{CISCOTIMESTAMP:ma… --- ## [Grok message pattern for log4j application log sample](https://discuss.elastic.co/t/grok-message-pattern-for-log4j-application-log-sample/90356)
**Author:** [@Bharath\_Raju](https://discuss.elastic.co/u/Bharath_Raju)\ **Replies:** 2\ **Last updated:** [June 22, 2017, 6:59am UTC](https://discuss.elastic.co/t/grok-message-pattern-for-log4j-application-log-sample/90356 "2017-06-22T06:59:24Z")
I have below set of log4j lines in the log file - 2017 Jun 13 01:56:51:303 GMT +5 BW.CustomerInfo Info \[BW-Core\] BWENGINE-300001 Process Engine version 5.10.0, build V80\_Hotfix08, 2013-5-27 2017 Jun 13 01:56:51:305 … --- ## [Logstash filter for start and end events repeating](https://discuss.elastic.co/t/logstash-filter-for-start-and-end-events-repeating/90036)
**Author:** [@pavangovinda](https://discuss.elastic.co/u/pavangovinda)\ **Replies:** 5\ **Last updated:** [June 22, 2017, 5:03am UTC](https://discuss.elastic.co/t/logstash-filter-for-start-and-end-events-repeating/90036 "2017-06-22T05:03:45Z")
Hi, I have a log in the below format.. (Not the complete log, only a sample) 2017-05-25T04:22:38.857 INFO - calling : Main.java method :main 2017-05-25T04:22:39.248 INFO - calling : PhiConfigParser.java method :getDocu… --- ## [Some logs are missing in Elasticsearch](https://discuss.elastic.co/t/some-logs-are-missing-in-elasticsearch/90214)
**Author:** [@ricky.chiu](https://discuss.elastic.co/u/ricky.chiu)\ **Replies:** 11\ **Last updated:** [June 21, 2017, 9:51am UTC](https://discuss.elastic.co/t/some-logs-are-missing-in-elasticsearch/90214 "2017-06-21T09:51:22Z")
Hi, We have a 3 node cluster ( 2 data nodes and 1 ingest node ) which is Centos 7.3, openjdk version "1.8.0\_121", Logstash 5.2 and Elasticsearch 5.2. We receives the log files every 5 minutes. We found that some logs… --- ## [LogStash JDBC plugin - Issues with last\_run\_metadata\_path](https://discuss.elastic.co/t/logstash-jdbc-plugin-issues-with-last-run-metadata-path/89137)
**Author:** [@mulgar](https://discuss.elastic.co/u/mulgar)\ **Replies:** 1\ **Last updated:** [June 22, 2017, 1:02am UTC](https://discuss.elastic.co/t/logstash-jdbc-plugin-issues-with-last-run-metadata-path/89137 "2017-06-22T01:02:08Z")
I'm having some problems with the last\_run\_metadata\_path in the LogStash JDBC Input plugin (ref: Jdbc input plugin | Logstash Reference \[8.11\] | Elastic ). I have the sql\_last\_value parameter working as expected previou… --- ## [Shipping collectd data to logstash](https://discuss.elastic.co/t/shipping-collectd-data-to-logstash/90327)
**Author:** [@mr\_chang7](https://discuss.elastic.co/u/mr_chang7)\ **Replies:** 2\ **Last updated:** [June 22, 2017, 12:30am UTC](https://discuss.elastic.co/t/shipping-collectd-data-to-logstash/90327 "2017-06-22T00:30:45Z")
We'd like to use collectd to ship the data from a node to logstash running in another node. But due to the license (GPL) issue we cannot use the 'network' plugin. Any other plugins we could try or what would you suggest … --- ## [Getting \_dateparsefailure](https://discuss.elastic.co/t/getting--dateparsefailure/89986)
**Author:** [@Amit2015](https://discuss.elastic.co/u/Amit2015)\ **Replies:** 8\ **Last updated:** [June 21, 2017, 8:29pm UTC](https://discuss.elastic.co/t/getting--dateparsefailure/89986 "2017-06-21T20:29:06Z")
Hi, I have date format like 20170616155531767-0500 which is unable to parse. Can you give me solution how can I parse date. I used date filter : date { locale =\> "en" match =\> \[ "REQ… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=958) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=960)