# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1020 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1021 --- ## [Yet another buffer question](https://discuss.elastic.co/t/yet-another-buffer-question/78259)
**Author:** [@jetnet](https://discuss.elastic.co/u/jetnet)\ **Replies:** 5\ **Last updated:** [March 12, 2017, 6:57am UTC](https://discuss.elastic.co/t/yet-another-buffer-question/78259 "2017-03-12T06:57:58Z")
Hello All, I'm indexing local log-files with Logstash + ES and encountered a "buffer" issue: the last messages get sent to ES only after sending Ctrl-C to the LS-process, e.g.: all logfiles are parsed: ES index - 96,52… --- ## [Logstash Gelf - Multiple sources](https://discuss.elastic.co/t/logstash-gelf-multiple-sources/78260)
**Author:** [@RSundareswaran](https://discuss.elastic.co/u/RSundareswaran)\ **Replies:** 0\ **Last updated:** [March 11, 2017, 9:24pm UTC](https://discuss.elastic.co/t/logstash-gelf-multiple-sources/78260 "2017-03-11T21:24:51Z")
Hi, I am trying to use gelf input plugin for some ESB logs with log4j2-GELF appender (just started ELK Stack for logging). I am successful in getting the logs through a configured port. As there are around 100 apps (wi… --- ## [How to load documents so you can query by day of week in PST](https://discuss.elastic.co/t/how-to-load-documents-so-you-can-query-by-day-of-week-in-pst/78061)
**Author:** [@dlam](https://discuss.elastic.co/u/dlam)\ **Replies:** 2\ **Last updated:** [March 11, 2017, 7:21pm UTC](https://discuss.elastic.co/t/how-to-load-documents-so-you-can-query-by-day-of-week-in-pst/78061 "2017-03-11T19:21:35Z")
hey i've been reading up and looks like elasticsearch/logstash will store dates as UTC by default. makes sense But here's the thing I'm trying to solve: I'm trying to do some analytics for an email marketing newslett… --- ## [\[SOLVED\] How to manage HUGE log messages sent to ELK stack over-the-wire?](https://discuss.elastic.co/t/solved-how-to-manage-huge-log-messages-sent-to-elk-stack-over-the-wire/78054)
**Author:** [@ZillaG](https://discuss.elastic.co/u/ZillaG)\ **Replies:** 2\ **Last updated:** [March 11, 2017, 6:07pm UTC](https://discuss.elastic.co/t/solved-how-to-manage-huge-log-messages-sent-to-elk-stack-over-the-wire/78054 "2017-03-11T18:07:37Z")
I've been using ELK for centralized logging service, and rsyslog as my shipper. Rsyslog wraps the log events in a JSON format. However, some of our application logs are HUGE, e.g., stack traces, and when Logstash recei… --- ## [Mongodb Oplog input plugin for logstash](https://discuss.elastic.co/t/mongodb-oplog-input-plugin-for-logstash/78240)
**Author:** [@sonpv1](https://discuss.elastic.co/u/sonpv1)\ **Replies:** 0\ **Last updated:** [March 11, 2017, 1:01pm UTC](https://discuss.elastic.co/t/mongodb-oplog-input-plugin-for-logstash/78240 "2017-03-11T13:01:32Z")
Hi Experts, I'm working on MongoDB Oplog task and I need to have a solution to transfer the recent data from MongoDB to Elasticsearch. I've just looked up on Logstash Input Plugins Page and I found nothing. It seems th… --- ## [Does logstash use http or transport client?](https://discuss.elastic.co/t/does-logstash-use-http-or-transport-client/78175)
**Author:** [@JonathanAaron](https://discuss.elastic.co/u/JonathanAaron)\ **Replies:** 1\ **Last updated:** [March 10, 2017, 6:11pm UTC](https://discuss.elastic.co/t/does-logstash-use-http-or-transport-client/78175 "2017-03-10T18:11:49Z")
Hey gang! Does logstash use http rest endpoints to insert data into elasticsearch or does it use Transport client? --- ## [X-Pack on Logstash Shipper](https://discuss.elastic.co/t/x-pack-on-logstash-shipper/78172)
**Author:** [@kopacko](https://discuss.elastic.co/u/kopacko)\ **Replies:** 4\ **Last updated:** [March 10, 2017, 5:08pm UTC](https://discuss.elastic.co/t/x-pack-on-logstash-shipper/78172 "2017-03-10T17:08:31Z")
I have several dedicated Logstash shippers who do not have Kibana or Elasticsearch installed. Can I or should I install X-pack on them? --- ## [Use first line of file for field names](https://discuss.elastic.co/t/use-first-line-of-file-for-field-names/78171)
**Author:** [@andre22](https://discuss.elastic.co/u/andre22)\ **Replies:** 0\ **Last updated:** [March 10, 2017, 4:47pm UTC](https://discuss.elastic.co/t/use-first-line-of-file-for-field-names/78171 "2017-03-10T16:47:00Z")
Hi, importing log files to analyze I realise that they have slightly differences in their structure, so i would have to create lots of patterns and conditions to match them all. But the files have a first line like thi… --- ## [Issue with index](https://discuss.elastic.co/t/issue-with-index/78028)
**Author:** [@bob-bza](https://discuss.elastic.co/u/bob-bza)\ **Replies:** 8\ **Last updated:** [March 10, 2017, 2:57pm UTC](https://discuss.elastic.co/t/issue-with-index/78028 "2017-03-10T14:57:00Z")
filebeat 1: - input\_type: log paths: - /var/log/ambari.log document\_type: ambari - input\_type: log paths: - /var/log/zookeeper.log document\_type: zookeeper tags: \["masternode"\] filebeat 2: - input\_typ… --- ## [Logstash 5.2.2 logstash-output-elastic plugin exception](https://discuss.elastic.co/t/logstash-5-2-2-logstash-output-elastic-plugin-exception/78002)
**Author:** [@Yahya\_Can](https://discuss.elastic.co/u/Yahya_Can)\ **Replies:** 1\ **Last updated:** [March 10, 2017, 2:43pm UTC](https://discuss.elastic.co/t/logstash-5-2-2-logstash-output-elastic-plugin-exception/78002 "2017-03-10T14:43:59Z")
Hi, I wanna move the data between csv to elastic in windows machine. When i run my config file with logstash.bat, i am getting following result. I checked plugin list. "logstash-output-elastic" is there. I updated plug… --- ## [Urldecode on kv results](https://discuss.elastic.co/t/urldecode-on-kv-results/78155)
**Author:** [@andre22](https://discuss.elastic.co/u/andre22)\ **Replies:** 3\ **Last updated:** [March 10, 2017, 2:39pm UTC](https://discuss.elastic.co/t/urldecode-on-kv-results/78155 "2017-03-10T14:39:59Z")
Hi, as hinted here I am dividing URI query parameters into multiple fields. Is there a way to run urldecode against the resulting fields without naming them specifically in the config? EDIT: this is the section from… --- ## [Docker SEBP/ELK not logging IP msgs](https://discuss.elastic.co/t/docker-sebp-elk-not-logging-ip-msgs/78153)
**Author:** [@ArnieNFW](https://discuss.elastic.co/u/ArnieNFW)\ **Replies:** 0\ **Last updated:** [March 10, 2017, 1:59pm UTC](https://discuss.elastic.co/t/docker-sebp-elk-not-logging-ip-msgs/78153 "2017-03-10T13:59:28Z")
I'm new to the stack but struggling for 2 weeks now and not making any progress. What I am trying to do is logging messages made by my IoT devices (EPS8266). I have the docker container running on the latest Ubuntu ser… --- ## [Map User Location with GeoIP and ELK](https://discuss.elastic.co/t/map-user-location-with-geoip-and-elk/78129)
**Author:** [@RustuGurkan](https://discuss.elastic.co/u/RustuGurkan)\ **Replies:** 3\ **Last updated:** [March 10, 2017, 1:50pm UTC](https://discuss.elastic.co/t/map-user-location-with-geoip-and-elk/78129 "2017-03-10T13:50:35Z")
Hi I'm having troubles with mapping user locations with GeoIP. Could someone help me please I followed the tutorial on: my config on /etc/logstash/12-apache.conf looks like this: geoip { source =\> "clientip… --- ## [Logstash docker container is trying to connect to elasticsearch](https://discuss.elastic.co/t/logstash-docker-container-is-trying-to-connect-to-elasticsearch/78146)
**Author:** [@jillesvangurp](https://discuss.elastic.co/u/jillesvangurp)\ **Replies:** 0\ **Last updated:** [March 10, 2017, 1:09pm UTC](https://discuss.elastic.co/t/logstash-docker-container-is-trying-to-connect-to-elasticsearch/78146 "2017-03-10T13:09:31Z")
I'm trying to get the logstash docker container up and running. So, when I run docker run docker.elastic.co/logstash/logstash:5.2.2 it tries to connect to elasticsearch out of the box even when my pipeline (layere… --- ## [Grok filtering in logstash for multiple defined patterns](https://discuss.elastic.co/t/grok-filtering-in-logstash-for-multiple-defined-patterns/77788)
**Author:** [@OucemaBellagha](https://discuss.elastic.co/u/OucemaBellagha)\ **Replies:** 5\ **Last updated:** [March 10, 2017, 12:42pm UTC](https://discuss.elastic.co/t/grok-filtering-in-logstash-for-multiple-defined-patterns/77788 "2017-03-10T12:42:27Z")
I am trying to filter my logs matching few patterns I have in the field "full\_log" e.g: E/vincinity/dholland\_view\_sql\_global/IN/Cluster\_Node/SSL-CACHE/Dsal1 F/vincinity/dholland\_view\_sql\_local/IN/Cluster\_Node3/SSL-CACH… --- ## [Catching JSON parse errors](https://discuss.elastic.co/t/catching-json-parse-errors/78083)
**Author:** [@elssar](https://discuss.elastic.co/u/elssar)\ **Replies:** 3\ **Last updated:** [March 10, 2017, 12:40pm UTC](https://discuss.elastic.co/t/catching-json-parse-errors/78083 "2017-03-10T12:40:52Z")
Hi, We have an old logstash system (v1.5) which had been running just fine until recently. Some of the messages are getting mangled, for reasons unknown, and the JSON filter keeps throwing parse fail errors. That cause… --- ## [Grok filter does not match message](https://discuss.elastic.co/t/grok-filter-does-not-match-message/78128)
**Author:** [@andre22](https://discuss.elastic.co/u/andre22)\ **Replies:** 1\ **Last updated:** [March 10, 2017, 11:24am UTC](https://discuss.elastic.co/t/grok-filter-does-not-match-message/78128 "2017-03-10T11:24:05Z")
Hi, when importing the following log lines, they are getting into Elasticsearch without being matched by grok: 2017-02-16 00:03:03 202.152.71.0 - www.host.ch GET /content/specialinterest var1=234&var3=876234 200 - - … --- ## [Persisted queue the uploaded logs are being sent again after restart](https://discuss.elastic.co/t/persisted-queue-the-uploaded-logs-are-being-sent-again-after-restart/78031)
**Author:** [@gall4rdo](https://discuss.elastic.co/u/gall4rdo)\ **Replies:** 1\ **Last updated:** [March 10, 2017, 11:08am UTC](https://discuss.elastic.co/t/persisted-queue-the-uploaded-logs-are-being-sent-again-after-restart/78031 "2017-03-10T11:08:22Z")
Hi, I have configured logstash with persistent queue, but when i restart the service, they send again the older logs, instead of checkpoint logs. It is a problem, because we have duplicated logs, and as long as they ar… --- ## [Newbie question: different numbers of URL parameters](https://discuss.elastic.co/t/newbie-question-different-numbers-of-url-parameters/77978)
**Author:** [@andre22](https://discuss.elastic.co/u/andre22)\ **Replies:** 2\ **Last updated:** [March 10, 2017, 9:56am UTC](https://discuss.elastic.co/t/newbie-question-different-numbers-of-url-parameters/77978 "2017-03-10T09:56:34Z")
Hi, I am new to to the ELK stack and managed to get the first data into elastic and perform some searches. Now I would like to work with some of my customized web server logs. Very probably this has been answered befor… --- ## [Azure Redis Input plugin](https://discuss.elastic.co/t/azure-redis-input-plugin/78113)
**Author:** [@Stefan\_Stift](https://discuss.elastic.co/u/Stefan_Stift)\ **Replies:** 0\ **Last updated:** [March 10, 2017, 8:59am UTC](https://discuss.elastic.co/t/azure-redis-input-plugin/78113 "2017-03-10T08:59:30Z")
I would really like to use the azure redis cache as source of my logstash input - maily for the reason that azure redis provides ssl. Anyone has done this? Expirience? Maybe a working plugin? --- ## [In what format shoould i index xml data in elasticsearch](https://discuss.elastic.co/t/in-what-format-shoould-i-index-xml-data-in-elasticsearch/77935)
**Author:** [@Pranil\_Ambule](https://discuss.elastic.co/u/Pranil_Ambule)\ **Replies:** 4\ **Last updated:** [March 10, 2017, 5:39am UTC](https://discuss.elastic.co/t/in-what-format-shoould-i-index-xml-data-in-elasticsearch/77935 "2017-03-10T05:39:18Z")
Im using logstash to index my xml file data into elasticsearch. Now my question is how to store my data so that it will be faster to be searched on. 1)Should i put all the file data in one field 2)Should i add different … --- ## [Logstash keeps on going down](https://discuss.elastic.co/t/logstash-keeps-on-going-down/77960)
**Author:** [@Raja1](https://discuss.elastic.co/u/Raja1)\ **Replies:** 3\ **Last updated:** [March 10, 2017, 4:50am UTC](https://discuss.elastic.co/t/logstash-keeps-on-going-down/77960 "2017-03-10T04:50:22Z")
Hi All, I have done the setup in my system by using docker image and able to pull the logs from my respective folder for the first time but later on logstash keeps shutting down automatically. Even i forcefully start … --- ## [Get substring of a field and use it in index name](https://discuss.elastic.co/t/get-substring-of-a-field-and-use-it-in-index-name/77999)
**Author:** [@ldms020255](https://discuss.elastic.co/u/ldms020255)\ **Replies:** 2\ **Last updated:** [March 10, 2017, 12:29am UTC](https://discuss.elastic.co/t/get-substring-of-a-field-and-use-it-in-index-name/77999 "2017-03-10T00:29:33Z")
Here's my config input { redis { data\_type =\> "list" key =\> "filebeat" host =\> "127.0.0.1" port =\> 6379 threads =\> 5 type =\> "log" } } filter { grok { … --- ## [Am I misunderstanding if/else statements?](https://discuss.elastic.co/t/am-i-misunderstanding-if-else-statements/77931)
**Author:** [@helium](https://discuss.elastic.co/u/helium)\ **Replies:** 2\ **Last updated:** [March 9, 2017, 10:41pm UTC](https://discuss.elastic.co/t/am-i-misunderstanding-if-else-statements/77931 "2017-03-09T22:41:35Z")
Hi, I have a bunch of logstash (2.4.0) nodes with this output config: output { if \[environment\] == "stage" { elasticsearch { hosts =\> \[ "elasticsearch-01" \] index =\> "l… --- ## [Date format: remove time, keep month-day-year](https://discuss.elastic.co/t/date-format-remove-time-keep-month-day-year/78053)
**Author:** [@vanian](https://discuss.elastic.co/u/vanian)\ **Replies:** 0\ **Last updated:** [March 9, 2017, 9:18pm UTC](https://discuss.elastic.co/t/date-format-remove-time-keep-month-day-year/78053 "2017-03-09T21:18:48Z")
I'm using Logstash to parse a CSV file which has fields in the following format: AssignmentDate (column header) 2/15/2016 (value) If I open the CSV with Sublime Text, this value is shown like this: Feb 15, 2016 I… --- ## [Splitting nested JSON array](https://discuss.elastic.co/t/splitting-nested-json-array/75259)
**Author:** [@mliu1212](https://discuss.elastic.co/u/mliu1212)\ **Replies:** 0\ **Last updated:** [February 16, 2017, 1:48am UTC](https://discuss.elastic.co/t/splitting-nested-json-array/75259 "2017-02-16T01:48:37Z")
Logstash gives me an errors if i try to split a record of "Only String and Array types are splittable". Could I get some help with my config file? \*\*Update: figured problem, removing info --- ## [Bad Password Characters in Logstash Config file](https://discuss.elastic.co/t/bad-password-characters-in-logstash-config-file/78036)
**Author:** [@ccampbell](https://discuss.elastic.co/u/ccampbell)\ **Replies:** 3\ **Last updated:** [March 9, 2017, 8:43pm UTC](https://discuss.elastic.co/t/bad-password-characters-in-logstash-config-file/78036 "2017-03-09T20:43:17Z")
In testing out a Windows Eventlog collection system using the Elastic stack, I discovered a curious problem with the password I was using for connecting through Xpack. I had used a system generated password that used sev… --- ## [Feature request: geo filter](https://discuss.elastic.co/t/feature-request-geo-filter/78041)
**Author:** [@cmk1523](https://discuss.elastic.co/u/cmk1523)\ **Replies:** 1\ **Last updated:** [March 9, 2017, 8:40pm UTC](https://discuss.elastic.co/t/feature-request-geo-filter/78041 "2017-03-09T20:40:19Z")
First off, this has nothing do with the geoip filter... I'd like to see a geo filter that given a lat & lon it produce meta information like: country, city, state, town, road, ... maybe even address! Much like Google A… --- ## [New tag applied to every document?](https://discuss.elastic.co/t/new-tag-applied-to-every-document/41929)
**Author:** [@Jerry\_Hoffmeister](https://discuss.elastic.co/u/Jerry_Hoffmeister)\ **Replies:** 9\ **Last updated:** [March 9, 2017, 8:23pm UTC](https://discuss.elastic.co/t/new-tag-applied-to-every-document/41929 "2017-03-09T20:23:35Z")
I've updated everything to the latest versions (logstash, elasticsearch, filebeat, kibana) and I see this tag added to every document: beats\_input\_codec\_plain\_applied Everything looks to be working but what does this … --- ## [Where is logstash.stdout for v5.2?](https://discuss.elastic.co/t/where-is-logstash-stdout-for-v5-2/78023)
**Author:** [@ZillaG](https://discuss.elastic.co/u/ZillaG)\ **Replies:** 2\ **Last updated:** [March 9, 2017, 8:05pm UTC](https://discuss.elastic.co/t/where-is-logstash-stdout-for-v5-2/78023 "2017-03-09T20:05:20Z")
I'm playing with v5.2, and have my output filter set to stdout using the rubydebug codec. However I don't see my events logged into /var/log/logstash/logstash.stdout. Did this location change? I looked in the /etc/logsta… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1019) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1021)