# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=579 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 580 --- ## [How to filter "select" queries from a log file that contains over 30000 lines with logstash](https://discuss.elastic.co/t/how-to-filter-select-queries-from-a-log-file-that-contains-over-30000-lines-with-logstash/173708)
**Author:** [@angham](https://discuss.elastic.co/u/angham)\ **Replies:** 0\ **Last updated:** [March 25, 2019, 9:40am UTC](https://discuss.elastic.co/t/how-to-filter-select-queries-from-a-log-file-that-contains-over-30000-lines-with-logstash/173708 "2019-03-25T09:40:48Z")
how to filter "select" queries from a log file that contains over 30000 lines with logstash exp: name: securityStore persistence provider classname: org.hibernate.ejb.HibernatePersistence 15:49:05,479 DEBUG \[org.hibe… --- ## [Grokking DNS entries/hostnames with underscores - less expensive pattern?](https://discuss.elastic.co/t/grokking-dns-entries-hostnames-with-underscores-less-expensive-pattern/173706)
**Author:** [@swedishmike](https://discuss.elastic.co/u/swedishmike)\ **Replies:** 0\ **Last updated:** [March 25, 2019, 9:37am UTC](https://discuss.elastic.co/t/grokking-dns-entries-hostnames-with-underscores-less-expensive-pattern/173706 "2019-03-25T09:37:20Z")
Hi all, I am currently working on getting some DNS logs into Elastic via Logstash. Grokking the logs with the HOSTNAME pattern works for most hosts. However, there are some hostnames or DNS entries that actually starts… --- ## [Logstash 6.6.2 unable to read postfix maillog and there is no error in the log](https://discuss.elastic.co/t/logstash-6-6-2-unable-to-read-postfix-maillog-and-there-is-no-error-in-the-log/173666)
**Author:** [@subramani\_ponnuswamy](https://discuss.elastic.co/u/subramani_ponnuswamy)\ **Replies:** 1\ **Last updated:** [March 25, 2019, 8:11am UTC](https://discuss.elastic.co/t/logstash-6-6-2-unable-to-read-postfix-maillog-and-there-is-no-error-in-the-log/173666 "2019-03-25T08:11:53Z")
I have configured the logstash.conf file and run there is no error found. but it does'nt read the log file. Kindly help to fix this. logstash.config file details below: input { file { type =\> "postfix" path =\> "d:\\m… --- ## [Using python to import to logstash](https://discuss.elastic.co/t/using-python-to-import-to-logstash/173623)
**Author:** [@mathurin68](https://discuss.elastic.co/u/mathurin68)\ **Replies:** 0\ **Last updated:** [March 24, 2019, 2:30am UTC](https://discuss.elastic.co/t/using-python-to-import-to-logstash/173623 "2019-03-24T02:30:31Z")
Using python to grab data from various sources, in json form, BUT need to normalize field names, and add tags. locations, etc. So I need to get it to logstash(I guess). Is the easiest way to get it to logstash using pyt… --- ## [Logstash input mongodb plgin](https://discuss.elastic.co/t/logstash-input-mongodb-plgin/173606)
**Author:** [@Pradyumna\_Jha](https://discuss.elastic.co/u/Pradyumna_Jha)\ **Replies:** 0\ **Last updated:** [March 23, 2019, 5:00pm UTC](https://discuss.elastic.co/t/logstash-input-mongodb-plgin/173606 "2019-03-23T17:00:03Z")
Is logstash mongo input plugin keeps on running? Can it fetch and read the newer data inserted in Mongo DB without being starting logstash again and again? If not then is there way to schedule this every second to pull l… --- ## [Elastic output stream](https://discuss.elastic.co/t/elastic-output-stream/173587)
**Author:** [@kpover](https://discuss.elastic.co/u/kpover)\ **Replies:** 3\ **Last updated:** [March 23, 2019, 3:24pm UTC](https://discuss.elastic.co/t/elastic-output-stream/173587 "2019-03-23T15:24:59Z")
Hello, Does elasticsearch support output streams similar to Graylog as shown here? http://docs.graylog.org/en/3.0/pages/streams.html The use case is forwarding specific events to a commercial SIEM solution. Happy for … --- ## [Logstash with jdbc plugin for mongoDB](https://discuss.elastic.co/t/logstash-with-jdbc-plugin-for-mongodb/173601)
**Author:** [@Pradyumna\_Jha](https://discuss.elastic.co/u/Pradyumna_Jha)\ **Replies:** 0\ **Last updated:** [March 23, 2019, 1:43pm UTC](https://discuss.elastic.co/t/logstash-with-jdbc-plugin-for-mongodb/173601 "2019-03-23T13:43:39Z")
Is there any way to data of mongoDb from logstash using jdbc input plugin. I need to schedule it also and need to use tracking column as to pick new records only. Please suggest. --- ## [Wrong logstash log time](https://discuss.elastic.co/t/wrong-logstash-log-time/173578)
**Author:** [@sahere37](https://discuss.elastic.co/u/sahere37)\ **Replies:** 1\ **Last updated:** [March 23, 2019, 1:10pm UTC](https://discuss.elastic.co/t/wrong-logstash-log-time/173578 "2019-03-23T13:10:22Z")
hi all, Notably due to daylight savings time, the system clock has been adjusted forward one hour but the logstash log time did not changed. In the below example the time of log is "16:38:43" which should be "17:38:43"… --- ## [Multiple pipelines with file output to same file](https://discuss.elastic.co/t/multiple-pipelines-with-file-output-to-same-file/173255)
**Author:** [@dbernard](https://discuss.elastic.co/u/dbernard)\ **Replies:** 1\ **Last updated:** [March 23, 2019, 7:07am UTC](https://discuss.elastic.co/t/multiple-pipelines-with-file-output-to-same-file/173255 "2019-03-23T07:07:33Z")
Currently I have one pipeline and events from filebeat are written to files with the file output filter. I am transitioning to multiple pipelines but during the transaction some hosts will write to the "default" pipelin… --- ## [How to print or store all skipped lines of a large log file?](https://discuss.elastic.co/t/how-to-print-or-store-all-skipped-lines-of-a-large-log-file/173154)
**Author:** [@satyaban](https://discuss.elastic.co/u/satyaban)\ **Replies:** 1\ **Last updated:** [March 23, 2019, 6:59am UTC](https://discuss.elastic.co/t/how-to-print-or-store-all-skipped-lines-of-a-large-log-file/173154 "2019-03-23T06:59:13Z")
Dear All, Some times due incorrect pattern of some lines of logs inside complete log files are skipped by logstash. It only sends all lines having pattern matched to grok syntax i have written. Actually my team want t… --- ## [Enrichment with recent logs](https://discuss.elastic.co/t/enrichment-with-recent-logs/173410)
**Author:** [@farsonic](https://discuss.elastic.co/u/farsonic)\ **Replies:** 4\ **Last updated:** [March 23, 2019, 6:09am UTC](https://discuss.elastic.co/t/enrichment-with-recent-logs/173410 "2019-03-23T06:09:40Z")
Hello, I have a logstash server receiving security events from BRO and also from a firewall. Upon ingest I've created a new log field that merges source-ip, source-port, dest-ip and dest-port into a single field called … --- ## [Pasrse data between repeated xml tags](https://discuss.elastic.co/t/pasrse-data-between-repeated-xml-tags/172270)
**Author:** [@sverma](https://discuss.elastic.co/u/sverma)\ **Replies:** 9\ **Last updated:** [March 23, 2019, 1:46am UTC](https://discuss.elastic.co/t/pasrse-data-between-repeated-xml-tags/172270 "2019-03-23T01:46:07Z")
I have a xml file that has repeated tags and i need to extract the fields between those tags , in below mentioned xml data I want to get the data between the job\_list tags: \ \ **Author:** [@kharvey](https://discuss.elastic.co/u/kharvey)\ **Replies:** 1\ **Last updated:** [March 23, 2019, 12:09am UTC](https://discuss.elastic.co/t/logstash-as-filebeat-passthrough/173561 "2019-03-23T00:09:52Z") Is it possible to pass filebeat logs straight through logstash without logstash doing any processing? My setup is as follows Client using filebeats =\> beats input on logstash-proxy =\> tcp output on logstash-proxy =\> tc… --- ## [Help identifying large message that is causing a Kafka RecordTooLargeException](https://discuss.elastic.co/t/help-identifying-large-message-that-is-causing-a-kafka-recordtoolargeexception/170961)
**Author:** [@jbehrends](https://discuss.elastic.co/u/jbehrends)\ **Replies:** 2\ **Last updated:** [March 22, 2019, 8:31pm UTC](https://discuss.elastic.co/t/help-identifying-large-message-that-is-causing-a-kafka-recordtoolargeexception/170961 "2019-03-22T20:31:24Z")
Hello! I'm running Logstash 6.6.0 on each node in Kubernetes cluster. The agent is using a file input plugin to read logs from all containers running on the host. I'm then using a Kafka output plugin to send these mes… --- ## [Missing logs after some time](https://discuss.elastic.co/t/missing-logs-after-some-time/173556)
**Author:** [@GiteshKhanna](https://discuss.elastic.co/u/GiteshKhanna)\ **Replies:** 0\ **Last updated:** [March 22, 2019, 7:32pm UTC](https://discuss.elastic.co/t/missing-logs-after-some-time/173556 "2019-03-22T19:32:48Z")
I am able to see the logs appearing in Kibana, but after some time the logs stop coming on Kibana. Any solution to this? --- ## [Time difference using ruby](https://discuss.elastic.co/t/time-difference-using-ruby/173549)
**Author:** [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Replies:** 2\ **Last updated:** [March 22, 2019, 7:10pm UTC](https://discuss.elastic.co/t/time-difference-using-ruby/173549 "2019-03-22T19:10:53Z")
I am trying to get a time difference between two time field using ruby filter but looks like one of the timefiled is empty or nill and that is giving me error ERROR\] ruby - Ruby exception occurred: undefined method \`-'… --- ## [An exception happened when converging configuration-logstash agent](https://discuss.elastic.co/t/an-exception-happened-when-converging-configuration-logstash-agent/173515)
**Author:** [@naveenrt23](https://discuss.elastic.co/u/naveenrt23)\ **Replies:** 1\ **Last updated:** [March 22, 2019, 4:29pm UTC](https://discuss.elastic.co/t/an-exception-happened-when-converging-configuration-logstash-agent/173515 "2019-03-22T16:29:33Z")
Hello, I have just started using logstash and I am not really sure what the stack trace means. Can someone help me? \[2019-03-22T09:52:37,183\]\[ERROR\]\[logstash.agent \] An exception happened when converging conf… --- ## [Elasticsearch plugin with SSL and not verified certificate](https://discuss.elastic.co/t/elasticsearch-plugin-with-ssl-and-not-verified-certificate/173520)
**Author:** [@elk\_user188](https://discuss.elastic.co/u/elk_user188)\ **Replies:** 0\ **Last updated:** [March 22, 2019, 3:51pm UTC](https://discuss.elastic.co/t/elasticsearch-plugin-with-ssl-and-not-verified-certificate/173520 "2019-03-22T15:51:32Z")
Hi there! Is there any way to have the Elasticsearch filter plugin work having Elasticsearch using SSL with a not verified certificate? I'd need something like the output option " ssl\_certificate\_verification =\> 'false… --- ## [Setup of logstash and filebeat on central server](https://discuss.elastic.co/t/setup-of-logstash-and-filebeat-on-central-server/173507)
**Author:** [@raffepaffe](https://discuss.elastic.co/u/raffepaffe)\ **Replies:** 0\ **Last updated:** [March 22, 2019, 2:39pm UTC](https://discuss.elastic.co/t/setup-of-logstash-and-filebeat-on-central-server/173507 "2019-03-22T14:39:18Z")
We are setting up elasticsearch, kibana, logstash and filebeats on a server to analyse log files from many applications. Due to reasons\* each application log file ens up in a separate directory on the server. We have abo… --- ## [Output Wildcard for Hostname](https://discuss.elastic.co/t/output-wildcard-for-hostname/173222)
**Author:** [@teamg](https://discuss.elastic.co/u/teamg)\ **Replies:** 2\ **Last updated:** [March 22, 2019, 2:26pm UTC](https://discuss.elastic.co/t/output-wildcard-for-hostname/173222 "2019-03-22T14:26:58Z")
This code works: output { if \[host\]\[name\] in \["lab4a.local.com", "lab4b.local.com"\] { elasticsearch { index =\> "lab4-%{+YYYY.MM.dd}" } } } Because I have multiple hosts with the hostname lab4{a,b,c,d,e...z} it'… --- ## [Logstash queue + kafka](https://discuss.elastic.co/t/logstash-queue-kafka/173494)
**Author:** [@paniccontrol](https://discuss.elastic.co/u/paniccontrol)\ **Replies:** 0\ **Last updated:** [March 22, 2019, 1:25pm UTC](https://discuss.elastic.co/t/logstash-queue-kafka/173494 "2019-03-22T13:25:34Z")
Does it make sense to put persistent queue if i already have a kafka in front ? I have logstash consuming my kafka. On this logstash i set up a queue of 50go, and everytime the queue get full. I don't understand why ? … --- ## [How to extract a specific string from a log after a match in grok filter?](https://discuss.elastic.co/t/how-to-extract-a-specific-string-from-a-log-after-a-match-in-grok-filter/173473)
**Author:** [@WarriorHarb](https://discuss.elastic.co/u/WarriorHarb)\ **Replies:** 4\ **Last updated:** [March 22, 2019, 1:20pm UTC](https://discuss.elastic.co/t/how-to-extract-a-specific-string-from-a-log-after-a-match-in-grok-filter/173473 "2019-03-22T13:20:34Z")
Hello, So i have a multi-lined log, each line has a format, after creating a grok filter that detects the keyword "callid" ,i would like to extract the ID right after that keyword and save it to a new field. here is a s… --- ## [Syslog Parser error - Viavi Gigflow](https://discuss.elastic.co/t/syslog-parser-error-viavi-gigflow/173465)
**Author:** [@ronanohart](https://discuss.elastic.co/u/ronanohart)\ **Replies:** 1\ **Last updated:** [March 22, 2019, 1:14pm UTC](https://discuss.elastic.co/t/syslog-parser-error-viavi-gigflow/173465 "2019-03-22T13:14:56Z")
I am trying to parse the following syslog message and keep getting a grokparsefailure message as sent from Gigaflow {"Application":"TCP/34056","Eventer":"172.0.0.1","Syn Type":"Destination","appid":427272,"bytes":152,"… --- ## [How to remove lines with specifics information from text files](https://discuss.elastic.co/t/how-to-remove-lines-with-specifics-information-from-text-files/173479)
**Author:** [@Ludovic9](https://discuss.elastic.co/u/Ludovic9)\ **Replies:** 3\ **Last updated:** [March 22, 2019, 1:05pm UTC](https://discuss.elastic.co/t/how-to-remove-lines-with-specifics-information-from-text-files/173479 "2019-03-22T13:05:00Z")
Hello, I'm using filebeat to redirect dns log to logstash, then elasticsearch. Here my configuration : filter { if "windows\_dnslog" in \[tags\] { grok { match =\> { "message" =\> "%{DATESTAMP:dns\_time} %{GREEDYDA… --- ## [Exclude Lines containing specific string from IIS Logs](https://discuss.elastic.co/t/exclude-lines-containing-specific-string-from-iis-logs/173464)
**Author:** [@nyarlath](https://discuss.elastic.co/u/nyarlath)\ **Replies:** 1\ **Last updated:** [March 22, 2019, 1:04pm UTC](https://discuss.elastic.co/t/exclude-lines-containing-specific-string-from-iis-logs/173464 "2019-03-22T13:04:26Z")
Hello all, I have a simple question, but i cannot find an answer that satisfies me. I am sending IIS 7.5 logs to logstash, but we have a monitoring tool that is doing a healthcheck every minute. The message line is li… --- ## [How to stop duplicate entries to output file created by output file plugin logstash?](https://discuss.elastic.co/t/how-to-stop-duplicate-entries-to-output-file-created-by-output-file-plugin-logstash/173470)
**Author:** [@satyaban](https://discuss.elastic.co/u/satyaban)\ **Replies:** 1\ **Last updated:** [March 22, 2019, 1:02pm UTC](https://discuss.elastic.co/t/how-to-stop-duplicate-entries-to-output-file-created-by-output-file-plugin-logstash/173470 "2019-03-22T13:02:46Z")
Dear All, I have log file having content \[29-Dec-2017 20:30:06\] - ||CompanyID = 65 \[30-Dec-2017 20:30:06\] - ||CompanyID = 61 I am using output file plugin for storing logs to an output file as below output { f… --- ## [Input logs from Azure BLOB using logstash and output to ES](https://discuss.elastic.co/t/input-logs-from-azure-blob-using-logstash-and-output-to-es/171459)
**Author:** [@Prabhu2430](https://discuss.elastic.co/u/Prabhu2430)\ **Replies:** 5\ **Last updated:** [March 22, 2019, 12:56pm UTC](https://discuss.elastic.co/t/input-logs-from-azure-blob-using-logstash-and-output-to-es/171459 "2019-03-22T12:56:19Z")
Hi All, I have been trying to pull the logs of my webapp which is stored in my azure storage account {BLOB}. I am using the logstash-input-azureblob plugin which i have already installed. \[root@AZEUSELKVM01 ~\]# /usr/s… --- ## [How to pull data from kafka to elasticsearch](https://discuss.elastic.co/t/how-to-pull-data-from-kafka-to-elasticsearch/173445)
**Author:** [@khergner](https://discuss.elastic.co/u/khergner)\ **Replies:** 2\ **Last updated:** [March 22, 2019, 12:53pm UTC](https://discuss.elastic.co/t/how-to-pull-data-from-kafka-to-elasticsearch/173445 "2019-03-22T12:53:22Z")
Hi everyone I want to send data from beat to kafka after kafka send logstash. Kafka role is message que here. I would like to use two beat but not created index based on elastic. Logstash input { kafka { bootstr… --- ## [Parse apache access log all the fields and draw the graph using status/response](https://discuss.elastic.co/t/parse-apache-access-log-all-the-fields-and-draw-the-graph-using-status-response/173247)
**Author:** [@mschofield](https://discuss.elastic.co/u/mschofield)\ **Replies:** 1\ **Last updated:** [March 22, 2019, 12:49pm UTC](https://discuss.elastic.co/t/parse-apache-access-log-all-the-fields-and-draw-the-graph-using-status-response/173247 "2019-03-22T12:49:59Z")
I need to filter status like 200 or response in kibana graph. But available fields like status/response not showing to filter in Discover. Logstash code : input { beats { port =\> "5044" } } filter { if \[type\] == … --- ## [Exclude path at .conf file](https://discuss.elastic.co/t/exclude-path-at-conf-file/173122)
**Author:** [@manosmustang](https://discuss.elastic.co/u/manosmustang)\ **Replies:** 4\ **Last updated:** [March 22, 2019, 11:19am UTC](https://discuss.elastic.co/t/exclude-path-at-conf-file/173122 "2019-03-22T11:19:49Z")
Hello, I have the below netdata.conf file and i want to exclude the path /var/log/messages \[root@centos conf.d\]# cat netdata.conf input { beats { port =\> 5044 } } output { elasticsearch { ho… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=578) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=580)