# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=841 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 842 --- ## [Logstash configuration error - 15-01-2018](https://discuss.elastic.co/t/logstash-configuration-error-15-01-2018/115486)
**Author:** [@SJN8](https://discuss.elastic.co/u/SJN8)\ **Replies:** 4\ **Last updated:** [January 15, 2018, 9:13am UTC](https://discuss.elastic.co/t/logstash-configuration-error-15-01-2018/115486 "2018-01-15T09:13:41Z")
Hi All below in my conf file , input { kafka { bootstrap\_servers =\> "sev1:9092 , sev2:9092 , sev3:9092" topics =\> \["mytopic\] } } The filter part of this file is commented out t… --- ## [Cannot upload to Elasticsearch via Logstash](https://discuss.elastic.co/t/cannot-upload-to-elasticsearch-via-logstash/115484)
**Author:** [@vrael](https://discuss.elastic.co/u/vrael)\ **Replies:** 2\ **Last updated:** [January 15, 2018, 7:19am UTC](https://discuss.elastic.co/t/cannot-upload-to-elasticsearch-via-logstash/115484 "2018-01-15T07:19:37Z")
Hi, I'm trying to upload a csv to logstash and all I get is this even with the verbose output option. Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties logstash 6.1.0 jruby 9… --- ## [Won't load a config file](https://discuss.elastic.co/t/wont-load-a-config-file/115422)
**Author:** [@tstapenhorst](https://discuss.elastic.co/u/tstapenhorst)\ **Replies:** 2\ **Last updated:** [January 14, 2018, 4:48am UTC](https://discuss.elastic.co/t/wont-load-a-config-file/115422 "2018-01-14T04:48:45Z")
I can't get my fantasy\_hockey.config file to load into ElasticSearch (version 6.1.1). Here is my config file followed by my launch statement followed by the error output. According to the error output message I'm missi… --- ## [Issue with logstash event feed](https://discuss.elastic.co/t/issue-with-logstash-event-feed/114692)
**Author:** [@Tal\_Bar\_Or](https://discuss.elastic.co/u/Tal_Bar_Or)\ **Replies:** 7\ **Last updated:** [January 14, 2018, 3:48pm UTC](https://discuss.elastic.co/t/issue-with-logstash-event-feed/114692 "2018-01-14T15:48:42Z")
Hello, I have Logstash, Elasticsearch (ver 6.1.1 recently updated ) getting the feed from Snort Syslog working for more than 6 months already, until yesterday, suddenly I noticed that I am not getting all alerts anymo… --- ## [Can we avoid duplicate records with fingerprint plugin or read input only once?](https://discuss.elastic.co/t/can-we-avoid-duplicate-records-with-fingerprint-plugin-or-read-input-only-once/115444)
**Author:** [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Replies:** 0\ **Last updated:** [January 14, 2018, 1:11pm UTC](https://discuss.elastic.co/t/can-we-avoid-duplicate-records-with-fingerprint-plugin-or-read-input-only-once/115444 "2018-01-14T13:11:22Z")
Hi All I have a input as http\_poller, pulling data from a http endpoint and output as a http The configuration file is working well. But the issue i have is, since logstash runs on continuous basis it goes and insert … --- ## [Picking the right charset for Filebeat](https://discuss.elastic.co/t/picking-the-right-charset-for-filebeat/115438)
**Author:** [@X\_Calibur](https://discuss.elastic.co/u/X_Calibur)\ **Replies:** 0\ **Last updated:** [January 14, 2018, 11:13am UTC](https://discuss.elastic.co/t/picking-the-right-charset-for-filebeat/115438 "2018-01-14T11:13:40Z")
Hi, In the beginning I received the following error from Logstash when attempting to send in Linux security logs: "Received an event that has a different character encoding than you configured". After checking the log… --- ## [Logstash start Gem::LoadError:](https://discuss.elastic.co/t/logstash-start-gem-loaderror/115424)
**Author:** [@stvoffutt](https://discuss.elastic.co/u/stvoffutt)\ **Replies:** 0\ **Last updated:** [January 14, 2018, 12:32am UTC](https://discuss.elastic.co/t/logstash-start-gem-loaderror/115424 "2018-01-14T00:32:51Z")
New to ELK here. I am following install instructions for Logstash on Ubuntu 16.04. Elasticsearch and Kibana are already installed and function properly. When I run DEBUG=1 bin/logstash -e 'input { stdin { } } output { st… --- ## [GeoIP2 without Coordinates in 2019?](https://discuss.elastic.co/t/geoip2-without-coordinates-in-2019/113725)
**Author:** [@MikeC0](https://discuss.elastic.co/u/MikeC0)\ **Replies:** 4\ **Last updated:** [January 6, 2018, 4:34am UTC](https://discuss.elastic.co/t/geoip2-without-coordinates-in-2019/113725 "2018-01-06T04:34:38Z")
Logstash depends to GeoIP2 for the coordinates in mapping. According to Maxmind, https://dev.maxmind.com/geoip/geoip2/downloadable/, they plan to discontinue the latitude and longitude coordinates in 2019. How does it… --- ## [Persisted Queue (Re-Play of existing Data)](https://discuss.elastic.co/t/persisted-queue-re-play-of-existing-data/111452)
**Author:** [@neu5ron](https://discuss.elastic.co/u/neu5ron)\ **Replies:** 6\ **Last updated:** [January 13, 2018, 12:10am UTC](https://discuss.elastic.co/t/persisted-queue-re-play-of-existing-data/111452 "2018-01-13T00:10:41Z")
I have about 15GBs worth of pesisted queue's that I need to "replay"/"re-run" through the pipeline. However, after a restart/reboot/etc logstash will not begin to re-ingest the data that is queued. All the data is still … --- ## [Aggregate-filter to create nested arrays](https://discuss.elastic.co/t/aggregate-filter-to-create-nested-arrays/115367)
**Author:** [@bosar](https://discuss.elastic.co/u/bosar)\ **Replies:** 0\ **Last updated:** [January 12, 2018, 9:20pm UTC](https://discuss.elastic.co/t/aggregate-filter-to-create-nested-arrays/115367 "2018-01-12T21:20:57Z")
I'm new to logstash and have been browsing the web for a couple of day for a solution for my problem to create an nested array for a company structure to load into elasticsearch. I want to create an array of companys wit… --- ## [Grok to parse CEF extension fields](https://discuss.elastic.co/t/grok-to-parse-cef-extension-fields/114328)
**Author:** [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Replies:** 5\ **Last updated:** [January 12, 2018, 8:04pm UTC](https://discuss.elastic.co/t/grok-to-parse-cef-extension-fields/114328 "2018-01-12T20:04:30Z")
Hello, I'm trying to create a grok pattern to parse the extension fields in CEF message from an antivirus server. My problem right now is that the same field can have different types of data, sometimes it is an intenge… --- ## [Logstash 6.1.1 running logstash.conf file is giving errors on Windows 10](https://discuss.elastic.co/t/logstash-6-1-1-running-logstash-conf-file-is-giving-errors-on-windows-10/115355)
**Author:** [@akron](https://discuss.elastic.co/u/akron)\ **Replies:** 0\ **Last updated:** [January 12, 2018, 7:48pm UTC](https://discuss.elastic.co/t/logstash-6-1-1-running-logstash-conf-file-is-giving-errors-on-windows-10/115355 "2018-01-12T19:48:40Z")
I recently installed ES onto my personal pc to test some of the functionality. I'm working on setting up an automatic process for pushing and pulling data from SQL using JDBC; however, after installing logstash (unzippin… --- ## [Pipeline aborted due to error {:pipeline\_id=\>"main", :exception=\>#\](https://discuss.elastic.co/t/pipeline-aborted-due-to-error-pipeline-id-main-exception-grok-patternerror-pattern-ciscotimestamptz-log-date-not-defined/109474)
**Author:** [@adam.wendling](https://discuss.elastic.co/u/adam.wendling)\ **Replies:** 4\ **Last updated:** [January 12, 2018, 6:59pm UTC](https://discuss.elastic.co/t/pipeline-aborted-due-to-error-pipeline-id-main-exception-grok-patternerror-pattern-ciscotimestamptz-log-date-not-defined/109474 "2018-01-12T18:59:11Z")
I am new to logstash. I am able to write and use simple .conf files and they work. I am trying a more complex and not getting very far. I am getting errors when trying to run logstash config. I ran with --debug and got t… --- ## [Issues connecting to logstash on windows server](https://discuss.elastic.co/t/issues-connecting-to-logstash-on-windows-server/114933)
**Author:** [@pranay](https://discuss.elastic.co/u/pranay)\ **Replies:** 1\ **Last updated:** [January 12, 2018, 6:00pm UTC](https://discuss.elastic.co/t/issues-connecting-to-logstash-on-windows-server/114933 "2018-01-12T18:00:51Z")
Hi, I am new to ELK world and started installing ELK stack... elasticsearch and Kibana are up and running and am able to login using the localhost url, but having issues while connecting to logstash.. pls help 0x7cc3f2f… --- ## [How to pull data data from 2 kafka topics using logstash and index the data in two separate index in elasticsearch](https://discuss.elastic.co/t/how-to-pull-data-data-from-2-kafka-topics-using-logstash-and-index-the-data-in-two-separate-index-in-elasticsearch/114977)
**Author:** [@kamalbeg](https://discuss.elastic.co/u/kamalbeg)\ **Replies:** 2\ **Last updated:** [January 12, 2018, 5:51pm UTC](https://discuss.elastic.co/t/how-to-pull-data-data-from-2-kafka-topics-using-logstash-and-index-the-data-in-two-separate-index-in-elasticsearch/114977 "2018-01-12T17:51:55Z")
I have a question regarding Kafka. I have create two topics called first\_topic and second\_topic I have a logstash-kafka.conf file input { kafka { bootstrap\_servers =\> "servera:9092" topics =\> \["first\_topic"\] } } o… --- ## [Can't convert the string datatype to ip](https://discuss.elastic.co/t/cant-convert-the-string-datatype-to-ip/115323)
**Author:** [@MartyA](https://discuss.elastic.co/u/MartyA)\ **Replies:** 2\ **Last updated:** [January 12, 2018, 5:38pm UTC](https://discuss.elastic.co/t/cant-convert-the-string-datatype-to-ip/115323 "2018-01-12T17:38:37Z")
mutate { convert =\> { "ip\_addr" =\> "ip" } } I tried using this. It gives me an error before it even starts indexing anything. I read something in a post from about 2 years ago and it said convert can't change speciali… --- ## [Curl -k query works but Logstash cannot query](https://discuss.elastic.co/t/curl-k-query-works-but-logstash-cannot-query/115176)
**Author:** [@chapmantrain](https://discuss.elastic.co/u/chapmantrain)\ **Replies:** 8\ **Last updated:** [January 12, 2018, 5:27pm UTC](https://discuss.elastic.co/t/curl-k-query-works-but-logstash-cannot-query/115176 "2018-01-12T17:27:21Z")
When I issue a query via a curl -k command to an https elasticsearch end point, I get the query results, however, using the elasticsearch input plugin in logstash, I cannot retrieve the same data. curl -k -u xxxx https:… --- ## [S3 Input Plugin randomly fails](https://discuss.elastic.co/t/s3-input-plugin-randomly-fails/115342)
**Author:** [@kpaulsenegain](https://discuss.elastic.co/u/kpaulsenegain)\ **Replies:** 1\ **Last updated:** [January 12, 2018, 5:18pm UTC](https://discuss.elastic.co/t/s3-input-plugin-randomly-fails/115342 "2018-01-12T17:18:02Z")
I'm using Logstash version 6.0.1 and S3 plugin logstash-input 3.2.0. I'm trying to process Web Server IIS Access logs files from a S3 bucket. The files are large 2.4 GB. The issue is that it works basically but random… --- ## [Logstash configure issue](https://discuss.elastic.co/t/logstash-configure-issue/114813)
**Author:** [@hsiuming](https://discuss.elastic.co/u/hsiuming)\ **Replies:** 1\ **Last updated:** [January 12, 2018, 5:06pm UTC](https://discuss.elastic.co/t/logstash-configure-issue/114813 "2018-01-12T17:06:18Z")
Hello we now have a data is two layer in kibana json form. In kibana we can see : { "\_index": "XXXXXX", "\_type": "pcap", "\_id": "XXXXXX", "\_version": X, . . . "layers": { "ip\_src": \[ "192.168.X.XX" \], . . … --- ## [Custom Initialisation of sql\_last\_value](https://discuss.elastic.co/t/custom-initialisation-of-sql-last-value/115027)
**Author:** [@jay-trivedi](https://discuss.elastic.co/u/jay-trivedi)\ **Replies:** 1\ **Last updated:** [January 12, 2018, 4:42pm UTC](https://discuss.elastic.co/t/custom-initialisation-of-sql-last-value/115027 "2018-01-12T16:42:07Z")
Hi there, I am trying to set up a Logstash pipeline with JDBC plug in. So far so good. I am trying to figure out if it is possible to initialise sql\_last\_value with a custom value. Thanks in advance :slight\_smile: --- ## [Help with parsing Kaspersky logs](https://discuss.elastic.co/t/help-with-parsing-kaspersky-logs/115329)
**Author:** [@Feedy](https://discuss.elastic.co/u/Feedy)\ **Replies:** 1\ **Last updated:** [January 12, 2018, 4:22pm UTC](https://discuss.elastic.co/t/help-with-parsing-kaspersky-logs/115329 "2018-01-12T16:22:34Z")
Not sure if this should be asked here or maybe Kibana so please let me know if this isn't the right place..... I've written a grok pattern using a Grok debugger tool and the tool gives me the proper named fields but whe… --- ## [Http\_poller input only new lines?](https://discuss.elastic.co/t/http-poller-input-only-new-lines/115058)
**Author:** [@eebee](https://discuss.elastic.co/u/eebee)\ **Replies:** 11\ **Last updated:** [January 12, 2018, 4:14pm UTC](https://discuss.elastic.co/t/http-poller-input-only-new-lines/115058 "2018-01-12T16:14:38Z")
I'm using the http\_poller Logstash input plugin to ingest a logfile into Elasticsearch. But every time it polls data from the logfile, it polls the whole file. Config file: input { http\_poller { urls =\> { te… --- ## [Getting hour value from @timestamp in a custom 'hour' field](https://discuss.elastic.co/t/getting-hour-value-from-timestamp-in-a-custom-hour-field/111168)
**Author:** [@EtienneDemers](https://discuss.elastic.co/u/EtienneDemers)\ **Replies:** 4\ **Last updated:** [January 12, 2018, 2:50pm UTC](https://discuss.elastic.co/t/getting-hour-value-from-timestamp-in-a-custom-hour-field/111168 "2018-01-12T14:50:09Z")
Hi all, I'm currently tring to get the hour of my document's timestamp in a custom 'hour' field. I did some research and found the following solution but the resulting value in the hour field is always 5 hour higher th… --- ## [Logstash 6.1.1 do not receive data from Filebeat 6.1.1: ERR Failed to connect](https://discuss.elastic.co/t/logstash-6-1-1-do-not-receive-data-from-filebeat-6-1-1-err-failed-to-connect/115305)
**Author:** [@dorinand](https://discuss.elastic.co/u/dorinand)\ **Replies:** 0\ **Last updated:** [January 12, 2018, 2:14pm UTC](https://discuss.elastic.co/t/logstash-6-1-1-do-not-receive-data-from-filebeat-6-1-1-err-failed-to-connect/115305 "2018-01-12T14:14:04Z")
I am trying to make filebeat communicate with logstash. It partially works. I have two nodes. First is filebeat-node and this is filebeat.yml: filebeat.prospectors: - type: log enabled: true paths: - /home/centos/logs/\*… --- ## [Logstash branding/identity](https://discuss.elastic.co/t/logstash-branding-identity/115301)
**Author:** [@arisbanach](https://discuss.elastic.co/u/arisbanach)\ **Replies:** 0\ **Last updated:** [January 12, 2018, 2:03pm UTC](https://discuss.elastic.co/t/logstash-branding-identity/115301 "2018-01-12T14:03:12Z")
I'm just curious if Elastic has discussed renaming Logstash since it isn't just about logs and doesn't actually stash them. I'm not sure if that was its original purpose, but I remember when I first started learning abou… --- ## [Logstash-grok unable to identify process from \`ps -ef/cmdline/metricbeat\`](https://discuss.elastic.co/t/logstash-grok-unable-to-identify-process-from-ps-ef-cmdline-metricbeat/115172)
**Author:** [@jamesliu\_london](https://discuss.elastic.co/u/jamesliu_london)\ **Replies:** 1\ **Last updated:** [January 12, 2018, 2:01pm UTC](https://discuss.elastic.co/t/logstash-grok-unable-to-identify-process-from-ps-ef-cmdline-metricbeat/115172 "2018-01-12T14:01:17Z")
I've used MetricBeat6.1 with Logstash6.1-grok. This is a grok attempt to identify all the processes and categorize them. These are mainly Identified \[java\_apps\],\[unidentified\_java\_apps\] and \[general unidentified apps\] … --- ## [Odd DNS filter behaviour prevents event processing](https://discuss.elastic.co/t/odd-dns-filter-behaviour-prevents-event-processing/115279)
**Author:** [@chrbraun](https://discuss.elastic.co/u/chrbraun)\ **Replies:** 0\ **Last updated:** [January 12, 2018, 11:13am UTC](https://discuss.elastic.co/t/odd-dns-filter-behaviour-prevents-event-processing/115279 "2018-01-12T11:13:03Z")
I recently upgraded my elastic Stack to 6.x.Since 2 days I noticed a weird behaviour which results in a significant amount of events being dropped. I have a DNS filter which does a reverse lookup on all internal IPs whi… --- ## [Clear index before update](https://discuss.elastic.co/t/clear-index-before-update/111837)
**Author:** [@anne.lhote](https://discuss.elastic.co/u/anne.lhote)\ **Replies:** 2\ **Last updated:** [January 12, 2018, 11:06am UTC](https://discuss.elastic.co/t/clear-index-before-update/111837 "2018-01-12T11:06:32Z")
Hello, I use Logstash to feed my ES from a MysQL database. Everything seems to work so far but the update. When a row is deleted on the MySQL database, the record is still present on the ES side. Is there any solution l… --- ## [Node stats api result after dropping the logs](https://discuss.elastic.co/t/node-stats-api-result-after-dropping-the-logs/115247)
**Author:** [@shwesinhan](https://discuss.elastic.co/u/shwesinhan)\ **Replies:** 0\ **Last updated:** [January 12, 2018, 9:14am UTC](https://discuss.elastic.co/t/node-stats-api-result-after-dropping-the-logs/115247 "2018-01-12T09:14:19Z")
hi all may i ask one thing. i use node stats api to check event counts and duration time for logstash pipeline performance. pipeline config eg is: filter { if "Exception" in \[message\] { grok { # exceptions … --- ## [Parsing in logstash using (grok,prune,multiline filter}](https://discuss.elastic.co/t/parsing-in-logstash-using-grok-prune-multiline-filter/115125)
**Author:** [@cilzzz](https://discuss.elastic.co/u/cilzzz)\ **Replies:** 2\ **Last updated:** [January 12, 2018, 10:33am UTC](https://discuss.elastic.co/t/parsing-in-logstash-using-grok-prune-multiline-filter/115125 "2018-01-12T10:33:09Z")
Hello everyone, i have an export of a script below that i need to parse through logstash. Job Name : save\_start\_config-sftp Job Status: Success (0) Schedule Name : daily\_save\_start\_config Us… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=840) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=842)