# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=624 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 625 --- ## [Skew in Logstash ingestion](https://discuss.elastic.co/t/skew-in-logstash-ingestion/162955)
**Author:** [@amitavmohanty01](https://discuss.elastic.co/u/amitavmohanty01)\ **Replies:** 0\ **Last updated:** [January 4, 2019, 12:28pm UTC](https://discuss.elastic.co/t/skew-in-logstash-ingestion/162955 "2019-01-04T12:28:24Z")
I have a pipeline: a kafka cluster of 2 nodes (3 ZK nodes) -\> 2 logstash nodes -\> elastic cluster I expected the 2 Logstash nodes to have almost equal load but the stats are very different from this expectation. --- ## [Jdbc input plugin: is it possible to let only run 1 sql query at once per pipeline?](https://discuss.elastic.co/t/jdbc-input-plugin-is-it-possible-to-let-only-run-1-sql-query-at-once-per-pipeline/162936)
**Author:** [@asp](https://discuss.elastic.co/u/asp)\ **Replies:** 0\ **Last updated:** [January 4, 2019, 10:31am UTC](https://discuss.elastic.co/t/jdbc-input-plugin-is-it-possible-to-let-only-run-1-sql-query-at-once-per-pipeline/162936 "2019-01-04T10:31:10Z")
Hi, I would like to use logstash to access our relational database and do some cyclic queries. It is planed to use multiple logstash pipelines. Each pipeline will only contain 1 sql statement which is fired against the… --- ## [I need a very important help “tag cloud”](https://discuss.elastic.co/t/i-need-a-very-important-help-tag-cloud/162935)
**Author:** [@angelogabeira](https://discuss.elastic.co/u/angelogabeira)\ **Replies:** 0\ **Last updated:** [January 4, 2019, 10:27am UTC](https://discuss.elastic.co/t/i-need-a-very-important-help-tag-cloud/162935 "2019-01-04T10:27:30Z")
I'm having a hard time in a few days. In my cloud of words the whole sentences appear. I already did the command below but it did not work. Can someone help me? I already tried this direct command in kibana PUT db2-… --- ## [Marking url as dead. Last error: \[LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError\] Elasticsearch Unreachable](https://discuss.elastic.co/t/marking-url-as-dead-last-error-logstash-hostunreachableerror-elasticsearch-unreachable/162761)
**Author:** [@arun5635](https://discuss.elastic.co/u/arun5635)\ **Replies:** 29\ **Last updated:** [January 4, 2019, 10:19am UTC](https://discuss.elastic.co/t/marking-url-as-dead-last-error-logstash-hostunreachableerror-elasticsearch-unreachable/162761 "2019-01-04T10:19:44Z")
Hi I am not able to send the data to my AWS -ES . I am getting the below error \[logstash.outputs.elasticsearch\] Marking url as dead. Last error: \[LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError\]… --- ## [Rotate Log Files at IST time not UTC](https://discuss.elastic.co/t/rotate-log-files-at-ist-time-not-utc/162791)
**Author:** [@abhishekvrm444](https://discuss.elastic.co/u/abhishekvrm444)\ **Replies:** 8\ **Last updated:** [January 4, 2019, 8:11am UTC](https://discuss.elastic.co/t/rotate-log-files-at-ist-time-not-utc/162791 "2019-01-04T08:11:00Z")
Hi, i want to rotate the logs that i am creating using logstash file output plugin and same is happening but there is one problem arising that my system has IST(Indian standard Time) but the logtsash is rotating logs ba… --- ## [How to split an event to multiple events?](https://discuss.elastic.co/t/how-to-split-an-event-to-multiple-events/162406)
**Author:** [@michaelhe](https://discuss.elastic.co/u/michaelhe)\ **Replies:** 2\ **Last updated:** [January 4, 2019, 7:08am UTC](https://discuss.elastic.co/t/how-to-split-an-event-to-multiple-events/162406 "2019-01-04T07:08:09Z")
I'm trying to parse zookeeper wchc output using logstash: 0x167892507c74d32 /zookeeper/cluster\_name/stores/31/alive /zookeeper/cluster\_name/stores/24/alive /zookeeper/cluster\_name/stores/32/alive for each session id… --- ## [Monitoring logstash](https://discuss.elastic.co/t/monitoring-logstash/162120)
**Author:** [@2501492a6edd02b1386b](https://discuss.elastic.co/u/2501492a6edd02b1386b)\ **Replies:** 2\ **Last updated:** [January 4, 2019, 7:02am UTC](https://discuss.elastic.co/t/monitoring-logstash/162120 "2019-01-04T07:02:53Z")
hello I am trying to configure monitoring for logstash via x-pack and faces the issue. When I execute logstash I see following error in logs: \[2018-12-26T12:21:15,309\]\[ERROR\]\[logstash.licensechecker.licensereader\] Una… --- ## [Error registering plugin for amazon\_es](https://discuss.elastic.co/t/error-registering-plugin-for-amazon-es/162796)
**Author:** [@arun5635](https://discuss.elastic.co/u/arun5635)\ **Replies:** 0\ **Last updated:** [January 3, 2019, 11:57am UTC](https://discuss.elastic.co/t/error-registering-plugin-for-amazon-es/162796 "2019-01-03T11:57:16Z")
Hi I am using amazon\_es output plugin , Im getting below error. \[2019-01-03T10:20:40,342\]\[ERROR\]\[logstash.pipeline \] Error registering plugin {:pipeline\_id=>"alb-accesslogs", :plugin=>"#LogStash::OutputDelegator:0… --- ## [Logstash Windows Service Filling up C:\\Windows\\Temp](https://discuss.elastic.co/t/logstash-windows-service-filling-up-c-windows-temp/162437)
**Author:** [@saravanan\_palanivel](https://discuss.elastic.co/u/saravanan_palanivel)\ **Replies:** 3\ **Last updated:** [January 4, 2019, 4:15am UTC](https://discuss.elastic.co/t/logstash-windows-service-filling-up-c-windows-temp/162437 "2019-01-04T04:15:40Z")
Hi Team, I'm running Logstash 6.5.0 as Windows service (and i installed that using NSSM), whenever Logstash service running in the background, it creating temp jar files into C:\\Windows\\Temp directory, eventually C:\\ dr… --- ## [Dead\_lettter\_queue plugin error and data processing behavior](https://discuss.elastic.co/t/dead-lettter-queue-plugin-error-and-data-processing-behavior/162879)
**Author:** [@Karthik\_Ramachandran](https://discuss.elastic.co/u/Karthik_Ramachandran)\ **Replies:** 0\ **Last updated:** [January 4, 2019, 12:55am UTC](https://discuss.elastic.co/t/dead-lettter-queue-plugin-error-and-data-processing-behavior/162879 "2019-01-04T00:55:08Z")
Situation: Logstash version (Docker Image 6.2.4) Multi-Pipeline mode is used for executing many pipelines dead-letter data processing pipeline is with id "main". We have 20+ pipelines apart from main The idea of main p… --- ## [I am sending a customized log file to filebeat, in which my time format is not able to come in date format type](https://discuss.elastic.co/t/i-am-sending-a-customized-log-file-to-filebeat-in-which-my-time-format-is-not-able-to-come-in-date-format-type/162479)
**Author:** [@SunilT](https://discuss.elastic.co/u/SunilT)\ **Replies:** 1\ **Last updated:** [January 3, 2019, 11:23pm UTC](https://discuss.elastic.co/t/i-am-sending-a-customized-log-file-to-filebeat-in-which-my-time-format-is-not-able-to-come-in-date-format-type/162479 "2019-01-03T23:23:32Z")
I am sending my data to elasticsearch through logstash from customized log file, with a "time2" field in it, i am not able to find a way how to convert time2 field into date type format. below is my log file: "score":{ … --- ## [Remove File After Read](https://discuss.elastic.co/t/remove-file-after-read/162809)
**Author:** [@OpSecMonkey](https://discuss.elastic.co/u/OpSecMonkey)\ **Replies:** 3\ **Last updated:** [January 3, 2019, 7:26pm UTC](https://discuss.elastic.co/t/remove-file-after-read/162809 "2019-01-03T19:26:23Z")
Hey I have a quick question. I was wondering if there is anyway to remove a file after it has been read by logstash. I have a feeling mine keeps hanging on the same file over and over but when I check the query there is… --- ## [Log4j-input-plugin buffer size in Logstash 6.x](https://discuss.elastic.co/t/log4j-input-plugin-buffer-size-in-logstash-6-x/161201)
**Author:** [@Max\_Oryschuk](https://discuss.elastic.co/u/Max_Oryschuk)\ **Replies:** 2\ **Last updated:** [January 3, 2019, 6:10pm UTC](https://discuss.elastic.co/t/log4j-input-plugin-buffer-size-in-logstash-6-x/161201 "2019-01-03T18:10:38Z")
Hi, After Logstash upgrade from 5.6.2 to 6.2.4 log4j input plugin started to introduce large log delivery delay. Seems like this's happening because Logstash is waiting for some specified amount of input data from log4… --- ## [Date filter not working to set timestamp and receive logs](https://discuss.elastic.co/t/date-filter-not-working-to-set-timestamp-and-receive-logs/161856)
**Author:** [@cazorino](https://discuss.elastic.co/u/cazorino)\ **Replies:** 2\ **Last updated:** [January 3, 2019, 4:21pm UTC](https://discuss.elastic.co/t/date-filter-not-working-to-set-timestamp-and-receive-logs/161856 "2019-01-03T16:21:49Z")
Hello community! I'm facing a problem when trying to receive logs and using their date field as timestamp, following is my log example: Dec 19 13:41:28 server-test smbd\_audit: DOMAIN+user.fake|127.0.0.1|general|open|ok… --- ## [Logstash - translate filter - enable encoding?](https://discuss.elastic.co/t/logstash-translate-filter-enable-encoding/162840)
**Author:** [@Francisca\_Lima](https://discuss.elastic.co/u/Francisca_Lima)\ **Replies:** 0\ **Last updated:** [January 3, 2019, 4:15pm UTC](https://discuss.elastic.co/t/logstash-translate-filter-enable-encoding/162840 "2019-01-03T16:15:25Z")
Hello, How can I enable encoding using translate filter? I have special characters in the field and they are not matching using translate filter in logstash, due to having special characters. Thank you. Francisca Lima … --- ## [Unit logstash.service entered failed state](https://discuss.elastic.co/t/unit-logstash-service-entered-failed-state/162804)
**Author:** [@arun5635](https://discuss.elastic.co/u/arun5635)\ **Replies:** 1\ **Last updated:** [January 3, 2019, 4:00pm UTC](https://discuss.elastic.co/t/unit-logstash-service-entered-failed-state/162804 "2019-01-03T16:00:11Z")
I am trying to start logstash via sudo service logstash start Redirecting to /bin/systemctl status logstash.service ● logstash.service - logstash Loaded: loaded (/etc/systemd/system/logstash.service; disabled; vendo… --- ## [Logstash sending /var/log/messages to Elastic](https://discuss.elastic.co/t/logstash-sending-var-log-messages-to-elastic/161727)
**Author:** [@Robert\_Fyffe](https://discuss.elastic.co/u/Robert_Fyffe)\ **Replies:** 1\ **Last updated:** [January 3, 2019, 3:41pm UTC](https://discuss.elastic.co/t/logstash-sending-var-log-messages-to-elastic/161727 "2019-01-03T15:41:48Z")
Hi Everyone, I am using the following 6.x stack: Filebeat -\> Logstash -\> Elastic -\> Kibana I am trying to configure filebeat/logstash to send a specific logfile only to Elastic. Right now it appears to be sending ever… --- ## [Differentiate between Nginx Access and Error logs without separate field from Filebeats](https://discuss.elastic.co/t/differentiate-between-nginx-access-and-error-logs-without-separate-field-from-filebeats/162834)
**Author:** [@duncOps](https://discuss.elastic.co/u/duncOps)\ **Replies:** 0\ **Last updated:** [January 3, 2019, 3:41pm UTC](https://discuss.elastic.co/t/differentiate-between-nginx-access-and-error-logs-without-separate-field-from-filebeats/162834 "2019-01-03T15:41:29Z")
Hi all, I'm running Filebeats in a container on DC/OS (a container orchestration platform similar to Kubernetes). These containers can access the logs of their parent VM, which means they can access the logs of any oth… --- ## [Influxdb problem](https://discuss.elastic.co/t/influxdb-problem/161292)
**Author:** [@abu.sayeed](https://discuss.elastic.co/u/abu.sayeed)\ **Replies:** 1\ **Last updated:** [January 3, 2019, 3:40pm UTC](https://discuss.elastic.co/t/influxdb-problem/161292 "2019-01-03T15:40:37Z")
\[2018-12-18T15:21:10,775\]\[WARN \]\[logstash.outputs.influxdb\] Non recoverable exception while writing to InfluxDB {:exception=\>#\ **Author:** [@abu.sayeed](https://discuss.elastic.co/u/abu.sayeed)\ **Replies:** 1\ **Last updated:** [January 3, 2019, 3:32pm UTC](https://discuss.elastic.co/t/logstash-main-pipeline-create-problems/161946 "2019-01-03T15:32:11Z") filter { if \[type\] == "syslog" { grok { match =\> \[ "message", "\<%{POSINT:syslog\_pri}\>%{SYSLOGTIMESTAMP:syslog\_timestamp} %{SYSLOGHOST:syslog\_hostname} %{DATA:syslog\_program}(?:\[%{POSINT:syslog\_pid}\])?: %{GREEDYDATA:sy… --- ## [Parsing apache access log file and extract timestamp](https://discuss.elastic.co/t/parsing-apache-access-log-file-and-extract-timestamp/162016)
**Author:** [@Eliot54](https://discuss.elastic.co/u/Eliot54)\ **Replies:** 1\ **Last updated:** [January 3, 2019, 3:30pm UTC](https://discuss.elastic.co/t/parsing-apache-access-log-file-and-extract-timestamp/162016 "2019-01-03T15:30:01Z")
I'm trying to use a custom log file with the following format in ELK : 109.70.81.5 - - \[01/Jul/1995:00:00:01 -0400\] "GET /history/apollo/ HTTP/1.0" 200 6245 However logstash /kibana keeps using my localhosts timestamp … --- ## [Filter messages based on tags](https://discuss.elastic.co/t/filter-messages-based-on-tags/162146)
**Author:** [@Mohammad\_Mousavi](https://discuss.elastic.co/u/Mohammad_Mousavi)\ **Replies:** 1\ **Last updated:** [January 3, 2019, 3:26pm UTC](https://discuss.elastic.co/t/filter-messages-based-on-tags/162146 "2019-01-03T15:26:54Z")
Hi I have 2 types of log inputs, nginx and iptabes. In filter section of logstash config file, I filter these messages and put a tag on them. my issue is that all of logstash messages ( input from filebeats ) forwards… --- ## [How do I set the logs for the logstash for windows?](https://discuss.elastic.co/t/how-do-i-set-the-logs-for-the-logstash-for-windows/162464)
**Author:** [@surajbs](https://discuss.elastic.co/u/surajbs)\ **Replies:** 1\ **Last updated:** [January 3, 2019, 3:12pm UTC](https://discuss.elastic.co/t/how-do-i-set-the-logs-for-the-logstash-for-windows/162464 "2019-01-03T15:12:49Z")
How do I set the logs for the logstash ? How can I set the Directory to write Logstash internal logs to ? --- ## [Is it possible to load complete text file in logstash](https://discuss.elastic.co/t/is-it-possible-to-load-complete-text-file-in-logstash/162580)
**Author:** [@asnar](https://discuss.elastic.co/u/asnar)\ **Replies:** 1\ **Last updated:** [January 3, 2019, 3:09pm UTC](https://discuss.elastic.co/t/is-it-possible-to-load-complete-text-file-in-logstash/162580 "2019-01-03T15:09:26Z")
Is it possible to load complete text file in logstash --- ## [How to json decode in logstash conf file?](https://discuss.elastic.co/t/how-to-json-decode-in-logstash-conf-file/162781)
**Author:** [@inandi](https://discuss.elastic.co/u/inandi)\ **Replies:** 1\ **Last updated:** [January 3, 2019, 3:06pm UTC](https://discuss.elastic.co/t/how-to-json-decode-in-logstash-conf-file/162781 "2019-01-03T15:06:13Z")
i have a dataset in which i have one json encoded data, I want to decode the data. Regards Gobinda Nandi --- ## [Docker logging driver gelf not working](https://discuss.elastic.co/t/docker-logging-driver-gelf-not-working/162175)
**Author:** [@kommineni24](https://discuss.elastic.co/u/kommineni24)\ **Replies:** 1\ **Last updated:** [December 27, 2018, 2:22am UTC](https://discuss.elastic.co/t/docker-logging-driver-gelf-not-working/162175 "2018-12-27T02:22:35Z")
Hello guys, Trying to send docker logs by using GELF logging driver to Logstash. Have configured udp and other details in logstash conf file. How to validate whether we are sending logs to kibana. --- ## [Calculate new fields dynamically in elasticsearch](https://discuss.elastic.co/t/calculate-new-fields-dynamically-in-elasticsearch/162758)
**Author:** [@balumurari1](https://discuss.elastic.co/u/balumurari1)\ **Replies:** 1\ **Last updated:** [January 3, 2019, 12:30pm UTC](https://discuss.elastic.co/t/calculate-new-fields-dynamically-in-elasticsearch/162758 "2019-01-03T12:30:40Z")
Hello, I am loading data from database into elasticsearch using logstash , during loading, does Elastic Search allows us to create Trigger-like functionality with our own rules to create new fields? Ex:- can we add n… --- ## [Logstash exports data multiple times to csv](https://discuss.elastic.co/t/logstash-exports-data-multiple-times-to-csv/162779)
**Author:** [@eblindeman](https://discuss.elastic.co/u/eblindeman)\ **Replies:** 0\ **Last updated:** [January 3, 2019, 10:09am UTC](https://discuss.elastic.co/t/logstash-exports-data-multiple-times-to-csv/162779 "2019-01-03T10:09:18Z")
Hello I'm trying to create an export using logstash (5.6.12) with the pipeline as written below. But unfortunately it exports everything continously in stead of everything just once. I start it with the following comma… --- ## [Issue with reading Salesforce data](https://discuss.elastic.co/t/issue-with-reading-salesforce-data/162760)
**Author:** [@Surasha](https://discuss.elastic.co/u/Surasha)\ **Replies:** 0\ **Last updated:** [January 3, 2019, 7:42am UTC](https://discuss.elastic.co/t/issue-with-reading-salesforce-data/162760 "2019-01-03T07:42:15Z")
//WARNING message : WARN -- : You are setting a key that conflicts with a built-in method Restforce::Mash#length defined in Hash. This can cause unexpected behavior when accessing the key as a property. You can still acc… --- ## [PostGIS Spatial Data import in ElasticSearch](https://discuss.elastic.co/t/postgis-spatial-data-import-in-elasticsearch/162741)
**Author:** [@Rahul\_Tandon](https://discuss.elastic.co/u/Rahul_Tandon)\ **Replies:** 3\ **Last updated:** [January 3, 2019, 7:11am UTC](https://discuss.elastic.co/t/postgis-spatial-data-import-in-elasticsearch/162741 "2019-01-03T07:11:17Z")
I am trying to import spatial data from postgres(POSTGIS extension) but facing issue while imporing spatial data. I am able to import non spatial data from postgres to elasticsearch using logstash. elastic search 6.5.4 … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=623) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=625)