# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=693 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 694 --- ## [If \[@metadata\]\[field\] {} doesnot work](https://discuss.elastic.co/t/if-metadata-field-doesnot-work/146681)
**Author:** [@vemula.pranavi](https://discuss.elastic.co/u/vemula.pranavi)\ **Replies:** 2\ **Last updated:** [August 30, 2018, 12:03pm UTC](https://discuss.elastic.co/t/if-metadata-field-doesnot-work/146681 "2018-08-30T12:03:16Z")
I'm trying to give syntax like 'if \[@metadata\]\[field\] exists then do this' in output output{ if \[@metadata\]\[field\] { elasticsearch { hosts =\> \["http://localhost:9200"\] index =\> "%{\[@metadata\]\[field\]}" } } } I us… --- ## [How to Restrict Logstash to process new records only after previous record is completely processed](https://discuss.elastic.co/t/how-to-restrict-logstash-to-process-new-records-only-after-previous-record-is-completely-processed/146320)
**Author:** [@Arjun](https://discuss.elastic.co/u/Arjun)\ **Replies:** 5\ **Last updated:** [August 30, 2018, 12:01pm UTC](https://discuss.elastic.co/t/how-to-restrict-logstash-to-process-new-records-only-after-previous-record-is-completely-processed/146320 "2018-08-30T12:01:18Z")
Hi Team, Whenever new record enters Logstash it should wait until previous record is completely processed only then new record should be processed by Logstash. Please suggest me how to configure this in Logstash. Than… --- ## [Logstash how to extract data from elasticsearch in real time](https://discuss.elastic.co/t/logstash-how-to-extract-data-from-elasticsearch-in-real-time/146660)
**Author:** [@kevin518](https://discuss.elastic.co/u/kevin518)\ **Replies:** 1\ **Last updated:** [August 30, 2018, 11:13am UTC](https://discuss.elastic.co/t/logstash-how-to-extract-data-from-elasticsearch-in-real-time/146660 "2018-08-30T11:13:47Z")
How does logstash extract data from elasticsearch in real time? How does logstash record the last time to extract data from elasticsearch? --- ## [Clone plugin documentation wrong or a bug?](https://discuss.elastic.co/t/clone-plugin-documentation-wrong-or-a-bug/142156)
**Author:** [@dennis300481](https://discuss.elastic.co/u/dennis300481)\ **Replies:** 5\ **Last updated:** [August 30, 2018, 10:56am UTC](https://discuss.elastic.co/t/clone-plugin-documentation-wrong-or-a-bug/142156 "2018-08-30T10:56:00Z")
I was trying to use the following config: filter { clone { add\_tag =\> \[ "cloned" \] } } This is not doing what i expect. My expectation is, that this will clone the current event and the cloned one has a tag "clone… --- ## [Logstash hostname as field](https://discuss.elastic.co/t/logstash-hostname-as-field/146662)
**Author:** [@EdBanzil](https://discuss.elastic.co/u/EdBanzil)\ **Replies:** 2\ **Last updated:** [August 30, 2018, 10:39am UTC](https://discuss.elastic.co/t/logstash-hostname-as-field/146662 "2018-08-30T10:39:14Z")
Hi all, Just wondering how can i get the logstash hostname as field in a clustered environment? all i can see is the hostname of beats. TIA --- ## [Not able to send data to Zabbix server](https://discuss.elastic.co/t/not-able-to-send-data-to-zabbix-server/146629)
**Author:** [@Rahulkumar](https://discuss.elastic.co/u/Rahulkumar)\ **Replies:** 4\ **Last updated:** [August 30, 2018, 9:45am UTC](https://discuss.elastic.co/t/not-able-to-send-data-to-zabbix-server/146629 "2018-08-30T09:45:43Z")
Hi, I am trying to parse the logs and sending it to Zabbix. I am using centOS. In logstash.yml file, I have configured only 1 setting in pipeline settig section: path.config: /etc/logstash/conf.d/\*.conf Everything e… --- ## [\_geoip\_lookup\_failure Error](https://discuss.elastic.co/t/geoip-lookup-failure-error/146348)
**Author:** [@Ira](https://discuss.elastic.co/u/Ira)\ **Replies:** 2\ **Last updated:** [August 30, 2018, 8:30am UTC](https://discuss.elastic.co/t/geoip-lookup-failure-error/146348 "2018-08-30T08:30:50Z")
Hi, I'm trying to add a geoip filter. My configuration file looks like this: input { beats { port =\> "5044" } } filter { grok{ match =\> { message =\> "%{TIMESTAMP\_ISO8601:timestamp}%{SPACE}\[%{DATA:thread}\]%{SPA… --- ## [Delete several documents in elastic with logstash?](https://discuss.elastic.co/t/delete-several-documents-in-elastic-with-logstash/146555)
**Author:** [@nicolasl](https://discuss.elastic.co/u/nicolasl)\ **Replies:** 1\ **Last updated:** [August 30, 2018, 8:00am UTC](https://discuss.elastic.co/t/delete-several-documents-in-elastic-with-logstash/146555 "2018-08-30T08:00:58Z")
Hello, I would like to delete multiple document with logstash in elasticsearch before sending the data to elastic. Why .. ? For example, I ingest a file into elastic with logstash. On the next day, I receive an updated… --- ## [Specifie error logged in logstash-plain.log](https://discuss.elastic.co/t/specifie-error-logged-in-logstash-plain-log/146473)
**Author:** [@wonki](https://discuss.elastic.co/u/wonki)\ **Replies:** 9\ **Last updated:** [August 30, 2018, 6:39am UTC](https://discuss.elastic.co/t/specifie-error-logged-in-logstash-plain-log/146473 "2018-08-30T06:39:12Z")
I thought my elastic stack runs well, but in logstash-plain.log file, the below error logged repeatebly. Is it the normal situation?? Again, there is no problem to fetch log data indexed in elasticsearch using filebeat… --- ## [Parse XML Using Logstash](https://discuss.elastic.co/t/parse-xml-using-logstash/146623)
**Author:** [@Vivek\_Sharma](https://discuss.elastic.co/u/Vivek_Sharma)\ **Replies:** 1\ **Last updated:** [August 30, 2018, 6:04am UTC](https://discuss.elastic.co/t/parse-xml-using-logstash/146623 "2018-08-30T06:04:39Z")
Hi I have a XML in below format: \ \ORD12345\ \V11111\ \ Filebeat picks up this xml and sends to logstash. In logstash, I … --- ## [IF Statement being ignored](https://discuss.elastic.co/t/if-statement-being-ignored/146612)
**Author:** [@CamTheMan](https://discuss.elastic.co/u/CamTheMan)\ **Replies:** 2\ **Last updated:** [August 30, 2018, 4:46am UTC](https://discuss.elastic.co/t/if-statement-being-ignored/146612 "2018-08-30T04:46:48Z")
Hello, I have a (simplified) filter rule like: filter { if "10.170.10.6" or "10.170.11.6" or "10.1.3.23" in \[host\] { grok { match =\> { "message" =\> \[ '^Accepted keyboard-interact… --- ## [There are no data output to es with no error](https://discuss.elastic.co/t/there-are-no-data-output-to-es-with-no-error/145004)
**Author:** [@kennymarx0](https://discuss.elastic.co/u/kennymarx0)\ **Replies:** 3\ **Last updated:** [August 30, 2018, 3:46am UTC](https://discuss.elastic.co/t/there-are-no-data-output-to-es-with-no-error/145004 "2018-08-30T03:46:30Z")
there are no data output to es with no error in logs. why haven't my data been imported? logs is below \[2018-08-18T19:42:13,315\]\[INFO \]\[logstash.outputs.elasticsearch\] Attempting to install template {:manage\_template… --- ## [Many input ports to templated destinations](https://discuss.elastic.co/t/many-input-ports-to-templated-destinations/146021)
**Author:** [@Will\_Weber](https://discuss.elastic.co/u/Will_Weber)\ **Replies:** 4\ **Last updated:** [August 30, 2018, 3:30am UTC](https://discuss.elastic.co/t/many-input-ports-to-templated-destinations/146021 "2018-08-30T03:30:04Z")
Hey all! Looking primarily for a recommendation or two on how one would recommend structuring a logstash config that is supposed to receive nearly 300 potentially very diverse sets of logs over the network that don't ne… --- ## [Back pressure and persistent queue](https://discuss.elastic.co/t/back-pressure-and-persistent-queue/146590)
**Author:** [@kernelpanic](https://discuss.elastic.co/u/kernelpanic)\ **Replies:** 0\ **Last updated:** [August 29, 2018, 11:27pm UTC](https://discuss.elastic.co/t/back-pressure-and-persistent-queue/146590 "2018-08-29T23:27:49Z")
Hello all, From the documentation on persistent queues it seems to be saying only the beats framework inputs are protected - if I want my non-beats inputs to be protected from back pressure (e.g. syslogs) then do I need… --- ## [Associating multiple log lines with previous line timestamp](https://discuss.elastic.co/t/associating-multiple-log-lines-with-previous-line-timestamp/146582)
**Author:** [@kraogv](https://discuss.elastic.co/u/kraogv)\ **Replies:** 0\ **Last updated:** [August 29, 2018, 8:57pm UTC](https://discuss.elastic.co/t/associating-multiple-log-lines-with-previous-line-timestamp/146582 "2018-08-29T20:57:27Z")
Hi, I am currently trying to associate lines after I encounter the time field in log lines to that time field. Sample log: 06-26-18 03:58:51 AM top - 03:58:51 up 84 days, 2:05, 0 users, load average: 0.01, 0.08, 0.12 … --- ## [Logstash conf not loading new version](https://discuss.elastic.co/t/logstash-conf-not-loading-new-version/143591)
**Author:** [@Rock-HOG](https://discuss.elastic.co/u/Rock-HOG)\ **Replies:** 3\ **Last updated:** [August 29, 2018, 7:07pm UTC](https://discuss.elastic.co/t/logstash-conf-not-loading-new-version/143591 "2018-08-29T19:07:46Z")
Hello, I have this issue with a conf file. I create the file and start laying out a filter. I then save and test it, everything is going well. Then ill add something like another grok filter, add a tag, or overwrite a f… --- ## [PQ Bytes Reporting](https://discuss.elastic.co/t/pq-bytes-reporting/146572)
**Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 0\ **Last updated:** [August 29, 2018, 6:51pm UTC](https://discuss.elastic.co/t/pq-bytes-reporting/146572 "2018-08-29T18:51:57Z")
PQ has never reported it's size in bytes since upgrading to 6.3, are there additional configurations that need to occur to get this number to report? This is a Windows Server installation, if that matters. --- ## [Logstash avro decode avro fails](https://discuss.elastic.co/t/logstash-avro-decode-avro-fails/146565)
**Author:** [@Prateek\_Bhati](https://discuss.elastic.co/u/Prateek_Bhati)\ **Replies:** 0\ **Last updated:** [August 29, 2018, 5:07pm UTC](https://discuss.elastic.co/t/logstash-avro-decode-avro-fails/146565 "2018-08-29T17:07:01Z")
An unexpected error occurred! {:error=\>#\, :backtrace=\>\["org/jruby/ext/stringio/StringIO.java:829:in read'", "/local/apps/rtdtuser/resources/elk/logstash-5.3.1/vendor/bundle/jruby… --- ## [401 Authorization Error with logstash elastic cloud module](https://discuss.elastic.co/t/401-authorization-error-with-logstash-elastic-cloud-module/146477)
**Author:** [@aashishgahlawat](https://discuss.elastic.co/u/aashishgahlawat)\ **Replies:** 2\ **Last updated:** [August 29, 2018, 3:03pm UTC](https://discuss.elastic.co/t/401-authorization-error-with-logstash-elastic-cloud-module/146477 "2018-08-29T15:03:47Z")
Official logstash elastic cloud module Official doc for starting with My logstash.yml looks like: # ------------ Cloud Settings --------------- # Define Elastic Cloud settings here. # Format of cloud.id is a base64 va… --- ## [Logstash fatal error: Not enough free disk space](https://discuss.elastic.co/t/logstash-fatal-error-not-enough-free-disk-space/143649)
**Author:** [@PolterFox](https://discuss.elastic.co/u/PolterFox)\ **Replies:** 12\ **Last updated:** [August 29, 2018, 2:02pm UTC](https://discuss.elastic.co/t/logstash-fatal-error-not-enough-free-disk-space/143649 "2018-08-29T14:02:44Z")
Hey guys, I have a problem with logstash, I have configured a persisted queue to process the events of logstash. This setup worked, but I increased the RAM from the system an restarted the server, since that Logstash th… --- ## [Logstash jdbc-input plugin recording sql last run time incorrectly](https://discuss.elastic.co/t/logstash-jdbc-input-plugin-recording-sql-last-run-time-incorrectly/146514)
**Author:** [@RishikeshDarandale](https://discuss.elastic.co/u/RishikeshDarandale)\ **Replies:** 0\ **Last updated:** [August 29, 2018, 10:50am UTC](https://discuss.elastic.co/t/logstash-jdbc-input-plugin-recording-sql-last-run-time-incorrectly/146514 "2018-08-29T10:50:04Z")
Here, I have MySQL server running in timezone America/Chicago (CDT/CST). I have verified this with following sql query SELECT @@global.time\_zone, @@session.time\_zone; My simple table looks like below: SELECT \* FROM br… --- ## [GEOPOINT not recognized](https://discuss.elastic.co/t/geopoint-not-recognized/146377)
**Author:** [@Alberto\_Sironi](https://discuss.elastic.co/u/Alberto_Sironi)\ **Replies:** 4\ **Last updated:** [August 29, 2018, 12:20pm UTC](https://discuss.elastic.co/t/geopoint-not-recognized/146377 "2018-08-29T12:20:58Z")
Hi guys, i'm using logstash 6.2.4 and i have some problem to upload my longitude and latitude point in kibana. I red a lot regarding this issue, but i still have some doubt. my logstash config file looks like: input { … --- ## [Import .tsv file through logstash](https://discuss.elastic.co/t/import-tsv-file-through-logstash/146489)
**Author:** [@dipti47](https://discuss.elastic.co/u/dipti47)\ **Replies:** 1\ **Last updated:** [August 29, 2018, 12:02pm UTC](https://discuss.elastic.co/t/import-tsv-file-through-logstash/146489 "2018-08-29T12:02:40Z")
Can i import .tsv file through logstash? --- ## [Mapping for \[kubernetes.labels.app\] tried to parse field \[app\] as object, but found a concrete value](https://discuss.elastic.co/t/mapping-for-kubernetes-labels-app-tried-to-parse-field-app-as-object-but-found-a-concrete-value/146490)
**Author:** [@Alaa.Mansour](https://discuss.elastic.co/u/Alaa.Mansour)\ **Replies:** 0\ **Last updated:** [August 29, 2018, 8:42am UTC](https://discuss.elastic.co/t/mapping-for-kubernetes-labels-app-tried-to-parse-field-app-as-object-but-found-a-concrete-value/146490 "2018-08-29T08:42:42Z")
Hello, I'm trying to run ELK stack on Cloud using kubernetes and ship logs and metrics to it with beats. Everything is installed using Kubernetes since everything is on the cloud and ELK is used to mintor kubernetes. C… --- ## [Logstash ruby filter - put all event fields to dynamic name nested field](https://discuss.elastic.co/t/logstash-ruby-filter-put-all-event-fields-to-dynamic-name-nested-field/146513)
**Author:** [@milkamar](https://discuss.elastic.co/u/milkamar)\ **Replies:** 0\ **Last updated:** [August 29, 2018, 10:43am UTC](https://discuss.elastic.co/t/logstash-ruby-filter-put-all-event-fields-to-dynamic-name-nested-field/146513 "2018-08-29T10:43:34Z")
Hey, to keep it short, I will simplify things a bit. Let's say I have two filebeats. Each filebeat has json output codec and outputs completely different set of fields. But, there is a rule - that messages can be pai… --- ## [Managing field names with spaces in Logstash](https://discuss.elastic.co/t/managing-field-names-with-spaces-in-logstash/146503)
**Author:** [@espogian](https://discuss.elastic.co/u/espogian)\ **Replies:** 0\ **Last updated:** [August 29, 2018, 10:01am UTC](https://discuss.elastic.co/t/managing-field-names-with-spaces-in-logstash/146503 "2018-08-29T10:01:38Z")
Hello, due to the way the skip\_header option is working in Logstash, when reading a CSV one has to define fields which have the exact same name of the header present in the CSV (see https://github.com/logstash-plugins/lo… --- ## [Change Field type from keyword to integer](https://discuss.elastic.co/t/change-field-type-from-keyword-to-integer/146497)
**Author:** [@PieterF](https://discuss.elastic.co/u/PieterF)\ **Replies:** 2\ **Last updated:** [August 29, 2018, 9:49am UTC](https://discuss.elastic.co/t/change-field-type-from-keyword-to-integer/146497 "2018-08-29T09:49:07Z")
Hi There, We use elasticsearch 5.6 with logstash,filebeat and kibana to search through the iis logs. We have a filebeat index that already is used but the Time Taken is not a number in Kibana. The "Time Taken propert… --- ## [Logstash 6.4.0 not parsing logs](https://discuss.elastic.co/t/logstash-6-4-0-not-parsing-logs/146472)
**Author:** [@espogian](https://discuss.elastic.co/u/espogian)\ **Replies:** 1\ **Last updated:** [August 29, 2018, 9:48am UTC](https://discuss.elastic.co/t/logstash-6-4-0-not-parsing-logs/146472 "2018-08-29T09:48:34Z")
Hi, moving from Logstash 6.3.0 to 6.4.0 with the exact same configuration and log files, I've noticed that the latter is not working. From Logstash logs I can see that the pipeline is starting, but no data is sent in ou… --- ## [Log size too large causes kibana to crash](https://discuss.elastic.co/t/log-size-too-large-causes-kibana-to-crash/146417)
**Author:** [@Derek\_Liu](https://discuss.elastic.co/u/Derek_Liu)\ **Replies:** 3\ **Last updated:** [August 29, 2018, 9:27am UTC](https://discuss.elastic.co/t/log-size-too-large-causes-kibana-to-crash/146417 "2018-08-29T09:27:12Z")
I am using packetbeat to monitor search queries and results on elasticsearch, then use logstash to send it to elasticsearch. Because the result of each search is too large, everytime when I try to load the logstash-\* ind… --- ## [Getting Oracle Error while running Logstash on RHEL7.5](https://discuss.elastic.co/t/getting-oracle-error-while-running-logstash-on-rhel7-5/146380)
**Author:** [@AshwiniSharma](https://discuss.elastic.co/u/AshwiniSharma)\ **Replies:** 1\ **Last updated:** [August 29, 2018, 9:20am UTC](https://discuss.elastic.co/t/getting-oracle-error-while-running-logstash-on-rhel7-5/146380 "2018-08-29T09:20:32Z")
Hi, I recently started using ELK. My configuration files were working correctly for logstash on windows setup. Now I have moved to RHEL7.5 and running the same setup but getting oracle error mentioned below: \[ERROR\]\[lo… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=692) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=694)