# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=474 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 475 --- ## [Logstash - ruby script to create array of nested fields](https://discuss.elastic.co/t/logstash-ruby-script-to-create-array-of-nested-fields/200312)
**Author:** [@Adithyaram](https://discuss.elastic.co/u/Adithyaram)\ **Replies:** 1\ **Last updated:** [September 19, 2019, 10:50pm UTC](https://discuss.elastic.co/t/logstash-ruby-script-to-create-array-of-nested-fields/200312 "2019-09-19T22:50:08Z")
Hello, I currently have the below JSON structure "Unit": { "Unit": \["QA", "FA"\], "QTY": \["2", "5"\], "ID": \["1", "2"\], "Item": \["Shirt", "Trousers"\], "Price": \["10", "50"\] } and want … --- ## [Removing part of message giving grokparsefailure](https://discuss.elastic.co/t/removing-part-of-message-giving-grokparsefailure/200234)
**Author:** [@jlim0930](https://discuss.elastic.co/u/jlim0930)\ **Replies:** 6\ **Last updated:** [September 19, 2019, 10:41pm UTC](https://discuss.elastic.co/t/removing-part-of-message-giving-grokparsefailure/200234 "2019-09-19T22:41:45Z")
while sending logs from a central syslog server it adds "Message forwarded from ...." in the message field. I added if "Message forwarded from" in \[message\] { mutate { gsub =\> \[ "message", "Message forwarded from ",… --- ## [Do I have to configure Logstash (to use GeoIP) if I am using a VM?](https://discuss.elastic.co/t/do-i-have-to-configure-logstash-to-use-geoip-if-i-am-using-a-vm/200313)
**Author:** [@Rosho](https://discuss.elastic.co/u/Rosho)\ **Replies:** 0\ **Last updated:** [September 19, 2019, 9:05pm UTC](https://discuss.elastic.co/t/do-i-have-to-configure-logstash-to-use-geoip-if-i-am-using-a-vm/200313 "2019-09-19T21:05:58Z")
Continuing the discussion from How does geoIP for private IPaddresses work?: The idea #1 of correlating IPs is difficult to implement because the public ones change dinamically. The idea #2: I need the range of IP… --- ## [Remove fields from documents based on their value](https://discuss.elastic.co/t/remove-fields-from-documents-based-on-their-value/200270)
**Author:** [@MMH](https://discuss.elastic.co/u/MMH)\ **Replies:** 7\ **Last updated:** [September 19, 2019, 7:54pm UTC](https://discuss.elastic.co/t/remove-fields-from-documents-based-on-their-value/200270 "2019-09-19T19:54:49Z")
I am using Logstash to read JSON data and load it to ElasticSearch. I am using a combination of update with doc\_as\_upsert to update existing documents and insert new. The specific application need is that in Elastic the… --- ## [Logstash pipeline output | duplicate messages ending up indexes](https://discuss.elastic.co/t/logstash-pipeline-output-duplicate-messages-ending-up-indexes/199878)
**Author:** [@deeps](https://discuss.elastic.co/u/deeps)\ **Replies:** 4\ **Last updated:** [September 19, 2019, 6:19pm UTC](https://discuss.elastic.co/t/logstash-pipeline-output-duplicate-messages-ending-up-indexes/199878 "2019-09-19T18:19:53Z")
In the /etc/logstash/conf.d/ directory I have configured 2 pipeline files to read messages from kafka topic/s and send it to data nodes in the cluster. (1.conf and 2.conf) \[root@ingest1 conf.d\]# egrep -w "topics|index" … --- ## [Kibana and Sonicwall sys logs](https://discuss.elastic.co/t/kibana-and-sonicwall-sys-logs/199905)
**Author:** [@RebeccaStanke](https://discuss.elastic.co/u/RebeccaStanke)\ **Replies:** 0\ **Last updated:** [September 18, 2019, 3:22am UTC](https://discuss.elastic.co/t/kibana-and-sonicwall-sys-logs/199905 "2019-09-18T03:22:14Z")
anyone have instructions on how to send sonicwall sys logs to Kibana for a newby? --- ## [Is logstash right choice for this?](https://discuss.elastic.co/t/is-logstash-right-choice-for-this/200280)
**Author:** [@plexcellmedia](https://discuss.elastic.co/u/plexcellmedia)\ **Replies:** 0\ **Last updated:** [September 19, 2019, 5:29pm UTC](https://discuss.elastic.co/t/is-logstash-right-choice-for-this/200280 "2019-09-19T17:29:27Z")
Hey, I'm having hard time here to go with right choice. I'm not sure if logstash is right way to go, hope you guys can help me out a bit. So, currently all our user tracking data is sent to elasticsearch directly which… --- ## [Ruby filter - compare a field against a mongodb and if exists the value add a new field](https://discuss.elastic.co/t/ruby-filter-compare-a-field-against-a-mongodb-and-if-exists-the-value-add-a-new-field/200279)
**Author:** [@Ricardo\_Pertuz](https://discuss.elastic.co/u/Ricardo_Pertuz)\ **Replies:** 0\ **Last updated:** [September 19, 2019, 5:20pm UTC](https://discuss.elastic.co/t/ruby-filter-compare-a-field-against-a-mongodb-and-if-exists-the-value-add-a-new-field/200279 "2019-09-19T17:20:48Z")
Hi, In my filebeat I'm adding a field called token, then I want to create a filter to check if the token value exists in my mongodb, if not then drop the entry and if so then mutate the token value to something else. Is… --- ## [Logstash output to file and s3 is different](https://discuss.elastic.co/t/logstash-output-to-file-and-s3-is-different/200272)
**Author:** [@shazbot](https://discuss.elastic.co/u/shazbot)\ **Replies:** 2\ **Last updated:** [September 19, 2019, 5:12pm UTC](https://discuss.elastic.co/t/logstash-output-to-file-and-s3-is-different/200272 "2019-09-19T17:12:49Z")
I'm trying to build something that would put logs into s3. In this case i'm working with netflow and my first step was to just get something out there. input { udp { port =\> 9995 codec =\> netflow } } outp… --- ## [Logstash JDBC input filter - Pagination](https://discuss.elastic.co/t/logstash-jdbc-input-filter-pagination/200274)
**Author:** [@sid1990](https://discuss.elastic.co/u/sid1990)\ **Replies:** 0\ **Last updated:** [September 19, 2019, 5:03pm UTC](https://discuss.elastic.co/t/logstash-jdbc-input-filter-pagination/200274 "2019-09-19T17:03:23Z")
Hi, I need a way to replicate what paging does , I'll simplify and explain only the problem part that we are struck on consider I have a table with ID's \[ 1 to 2000 \] and \[4000 - 1M\] ( ID's from 2001 to 3999 are not… --- ## [Fresh instalation - Logstash cannot connect to ElasticSearch](https://discuss.elastic.co/t/fresh-instalation-logstash-cannot-connect-to-elasticsearch/200085)
**Author:** [@MMH](https://discuss.elastic.co/u/MMH)\ **Replies:** 6\ **Last updated:** [September 19, 2019, 3:39pm UTC](https://discuss.elastic.co/t/fresh-instalation-logstash-cannot-connect-to-elasticsearch/200085 "2019-09-19T15:39:19Z")
Hi all, I installed Elasticsearch, Kibana, and Logstash from zip files on Windows. Elasticsearch seems to work - I can connect via Kibana and insert documents to the index. My browser returns JSON response when directe… --- ## [Why does logstash do not appear in the "Managament" option?](https://discuss.elastic.co/t/why-does-logstash-do-not-appear-in-the-managament-option/200075)
**Author:** [@Rosho](https://discuss.elastic.co/u/Rosho)\ **Replies:** 3\ **Last updated:** [September 19, 2019, 3:09pm UTC](https://discuss.elastic.co/t/why-does-logstash-do-not-appear-in-the-managament-option/200075 "2019-09-19T15:09:12Z")
Hi I have just viewed a tutorial about Logstash https://www.elastic.co/fr/webinars/getting-started-logstash?baymax=default&elektra=docs&storm=top-video I am using a VM to test the X-pack capabilities and the GeoIP wit… --- ## [Dateparsefailure on logstash](https://discuss.elastic.co/t/dateparsefailure-on-logstash/200235)
**Author:** [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Replies:** 2\ **Last updated:** [September 19, 2019, 2:48pm UTC](https://discuss.elastic.co/t/dateparsefailure-on-logstash/200235 "2019-09-19T14:48:40Z")
I am pulling data from sql and I have date field I am still getting this error on output. and can't figure out where this is coming from. "tags" =\> \[ \[0\] "\_dateparsefailure" \], I try without a… --- ## [Data from CSV is not inserting to Elastic search using Logstash](https://discuss.elastic.co/t/data-from-csv-is-not-inserting-to-elastic-search-using-logstash/200176)
**Author:** [@SudevPK](https://discuss.elastic.co/u/SudevPK)\ **Replies:** 1\ **Last updated:** [September 19, 2019, 2:39pm UTC](https://discuss.elastic.co/t/data-from-csv-is-not-inserting-to-elastic-search-using-logstash/200176 "2019-09-19T14:39:34Z")
Data is not Inserting to Elastic search using LogStash Config input { file { path =\> \["C:\\PR Files\\Test\\Account1.csv"\] start\_position =\> "beginning" sincedb\_path =\> "nul" } } filter { csv { separator =\> "|" columns =\> … --- ## [Logstash only ingesting data when shutdown/closing files](https://discuss.elastic.co/t/logstash-only-ingesting-data-when-shutdown-closing-files/200067)
**Author:** [@edster](https://discuss.elastic.co/u/edster)\ **Replies:** 5\ **Last updated:** [September 19, 2019, 2:20pm UTC](https://discuss.elastic.co/t/logstash-only-ingesting-data-when-shutdown-closing-files/200067 "2019-09-19T14:20:02Z")
My logstash config file seems to running successfully, however, it doesn't pull in any data. It isn't until I manually shutdown the process using 'Ctrl+C', that it pulls in data. This doesn't seem to normal behavior. Is … --- ## [Logstash exposure on port 12201 (gelf) permission denied](https://discuss.elastic.co/t/logstash-exposure-on-port-12201-gelf-permission-denied/200207)
**Author:** [@Edwin\_Istin](https://discuss.elastic.co/u/Edwin_Istin)\ **Replies:** 0\ **Last updated:** [September 19, 2019, 1:19pm UTC](https://discuss.elastic.co/t/logstash-exposure-on-port-12201-gelf-permission-denied/200207 "2019-09-19T13:19:57Z")
I deployed the ELK certified by Bitnami from GCP market place. My purpose is to receive data from my micro-services sending json to my Logstash pipeline. But when I try with curl (on localhost after use\_elk script) to PO… --- ## [How to filter required word from unstructured data (free text) to store into fileds](https://discuss.elastic.co/t/how-to-filter-required-word-from-unstructured-data-free-text-to-store-into-fileds/200124)
**Author:** [@Ennr](https://discuss.elastic.co/u/Ennr)\ **Replies:** 7\ **Last updated:** [September 19, 2019, 11:58am UTC](https://discuss.elastic.co/t/how-to-filter-required-word-from-unstructured-data-free-text-to-store-into-fileds/200124 "2019-09-19T11:58:45Z")
how to filter required word from unstructured data (free text) to store into fileds (i.e. name, address, persons,date etc.... ) here is the sample text : Note : this text is in German language. Öffentliche Bekanntmac… --- ## [How to update ES\_index data from sql server using log-stash Automatically](https://discuss.elastic.co/t/how-to-update-es-index-data-from-sql-server-using-log-stash-automatically/199642)
**Author:** [@rameshkr1994](https://discuss.elastic.co/u/rameshkr1994)\ **Replies:** 3\ **Last updated:** [September 19, 2019, 10:28am UTC](https://discuss.elastic.co/t/how-to-update-es-index-data-from-sql-server-using-log-stash-automatically/199642 "2019-09-19T10:28:28Z")
Dear All. once more time come up here with some little complex but not impossible issue. i am trying to fetch only changed /updated data from sql server to Elastic search index using log stash tools but i am getting er… --- ## [Data Lost while import data in elastic search using logstash](https://discuss.elastic.co/t/data-lost-while-import-data-in-elastic-search-using-logstash/200177)
**Author:** [@chintanvasani](https://discuss.elastic.co/u/chintanvasani)\ **Replies:** 0\ **Last updated:** [September 19, 2019, 10:20am UTC](https://discuss.elastic.co/t/data-lost-while-import-data-in-elastic-search-using-logstash/200177 "2019-09-19T10:20:46Z")
I am using logstash 5.5.0 and elastic search 5.5.0. I am trying to import data from txt file with | separator to elastic search. The total number of records in txt file are 2139406 approx but while importing the data t… --- ## [Error sending to RabbitMQ with priority from event - cannot convert instance of class org.jruby.specialized.RubyArrayOneObject to class java.lang.Integer](https://discuss.elastic.co/t/error-sending-to-rabbitmq-with-priority-from-event-cannot-convert-instance-of-class-org-jruby-specialized-rubyarrayoneobject-to-class-java-lang-integer/199965)
**Author:** [@Stuart\_Moore](https://discuss.elastic.co/u/Stuart_Moore)\ **Replies:** 4\ **Last updated:** [September 19, 2019, 10:09am UTC](https://discuss.elastic.co/t/error-sending-to-rabbitmq-with-priority-from-event-cannot-convert-instance-of-class-org-jruby-specialized-rubyarrayoneobject-to-class-java-lang-integer/199965 "2019-09-19T10:09:02Z")
I am trying to send a logstash event to RabbitMq with a priority from the event, and getting an error. The same code works correctly when I don't send priority. The code that fails: rabbitmq { arguments =\> … --- ## [Multiple logs in one document](https://discuss.elastic.co/t/multiple-logs-in-one-document/200161)
**Author:** [@KOTOXJle6](https://discuss.elastic.co/u/KOTOXJle6)\ **Replies:** 0\ **Last updated:** [September 19, 2019, 9:04am UTC](https://discuss.elastic.co/t/multiple-logs-in-one-document/200161 "2019-09-19T09:04:27Z")
Hello! Im trying to add Squid and MS Exchange logs to my ELK cluster. We need to store information about where users go on the Internet and user mail forwarding logs for the last 180 days. The problem with squid is tha… --- ## [When logstash is down , app container fails to start](https://discuss.elastic.co/t/when-logstash-is-down-app-container-fails-to-start/199195)
**Author:** [@markrity](https://discuss.elastic.co/u/markrity)\ **Replies:** 4\ **Last updated:** [September 19, 2019, 9:03am UTC](https://discuss.elastic.co/t/when-logstash-is-down-app-container-fails-to-start/199195 "2019-09-19T09:03:24Z")
Hi I run couple of micro-services that write logs to Logstash on ECS, Unfortunately if Logstash happens to be unavailable, any app-Task container that writes to logstash fails to startup. How to handle this kind of fai… --- ## [Why log stash is tracking columns other file.conf](https://discuss.elastic.co/t/why-log-stash-is-tracking-columns-other-file-conf/200154)
**Author:** [@rameshkr1994](https://discuss.elastic.co/u/rameshkr1994)\ **Replies:** 0\ **Last updated:** [September 19, 2019, 8:50am UTC](https://discuss.elastic.co/t/why-log-stash-is-tracking-columns-other-file-conf/200154 "2019-09-19T08:50:36Z")
Dear All. i recently run tracking columns query with Logstash.conf file but when i tried with other logstash1.conf file then it is through error why. i having difficulty to start my POC. Failed to execute action {:act… --- ## [Security patches in docker container?](https://discuss.elastic.co/t/security-patches-in-docker-container/200152)
**Author:** [@asp](https://discuss.elastic.co/u/asp)\ **Replies:** 0\ **Last updated:** [September 19, 2019, 8:48am UTC](https://discuss.elastic.co/t/security-patches-in-docker-container/200152 "2019-09-19T08:48:23Z")
Hi, I am using the official docker container. I am putting my pipeline configuration inside the image by creating a new image based on yours. What I am a bit curious about is the fact, that when I run yum update inside… --- ## [Not able to create multiple index through logstash 7.3](https://discuss.elastic.co/t/not-able-to-create-multiple-index-through-logstash-7-3/199754)
**Author:** [@manjeetsingh](https://discuss.elastic.co/u/manjeetsingh)\ **Replies:** 8\ **Last updated:** [September 19, 2019, 8:38am UTC](https://discuss.elastic.co/t/not-able-to-create-multiple-index-through-logstash-7-3/199754 "2019-09-19T08:38:33Z")
Hi Team, I have a setup of dockerized elk setup of version 7.3.1 (Elasticsearch, logstash, Kibana, Filebeat). ELK is running on docker only. I have installed filebeat package on same machine and provide input to logsta… --- ## [Logstash is not creating index to Elasticsearch version - 7.3.1](https://discuss.elastic.co/t/logstash-is-not-creating-index-to-elasticsearch-version-7-3-1/200115)
**Author:** [@ashish\_kumar1490](https://discuss.elastic.co/u/ashish_kumar1490)\ **Replies:** 4\ **Last updated:** [September 19, 2019, 6:24am UTC](https://discuss.elastic.co/t/logstash-is-not-creating-index-to-elasticsearch-version-7-3-1/200115 "2019-09-19T06:24:57Z")
I have ELK stack of version 7.3.1 .I am pasting my JSON using Logstash input { file { #path =\> \["D:\\ashishdoc.json"\] path =\> \["D:\\Sentient\\elk\_upgrade\\logstash-7.3.1\\logstash-7.3.1\\ashishdoc.json"\] start\_positio… --- ## [Unable to parse xml file](https://discuss.elastic.co/t/unable-to-parse-xml-file/200118)
**Author:** [@anshi8](https://discuss.elastic.co/u/anshi8)\ **Replies:** 0\ **Last updated:** [September 19, 2019, 6:05am UTC](https://discuss.elastic.co/t/unable-to-parse-xml-file/200118 "2019-09-19T06:05:16Z")
Hi, I am new to elastic stack . I am parsing my xml file to elasticsearch via logstash . But is unable to get any output neither in logstash nor kibana. I am using version 7.2.0 on windows 8. can anyone please help? T… --- ## [Unable to parse the date in syslog log message](https://discuss.elastic.co/t/unable-to-parse-the-date-in-syslog-log-message/199835)
**Author:** [@sai\_kiran1](https://discuss.elastic.co/u/sai_kiran1)\ **Replies:** 2\ **Last updated:** [September 19, 2019, 5:28am UTC](https://discuss.elastic.co/t/unable-to-parse-the-date-in-syslog-log-message/199835 "2019-09-19T05:28:12Z")
I am trying to parse the data in my Syslog log message using different patterns like %SYSLOGTIMESTAMP:syslog\_timestamp} , %{MONTH} +%{MONTHDAY} %{TIME} but am unable to parse the message. Message I am trying to parse "s… --- ## [Logstash is not pick up updated logs in newly created Log file with a same name after backup](https://discuss.elastic.co/t/logstash-is-not-pick-up-updated-logs-in-newly-created-log-file-with-a-same-name-after-backup/199620)
**Author:** [@ldeepak](https://discuss.elastic.co/u/ldeepak)\ **Replies:** 2\ **Last updated:** [September 19, 2019, 5:14am UTC](https://discuss.elastic.co/t/logstash-is-not-pick-up-updated-logs-in-newly-created-log-file-with-a-same-name-after-backup/199620 "2019-09-19T05:14:06Z")
Hi Team, Please help me on below topic. Logstash is not able to read updated logs after creating a new log file with the same and putting old log file as a backup file, this issue i had faced in windows. please find b… --- ## [How can i write to logstash without filebeat/agent?](https://discuss.elastic.co/t/how-can-i-write-to-logstash-without-filebeat-agent/200088)
**Author:** [@rajkumar.m](https://discuss.elastic.co/u/rajkumar.m)\ **Replies:** 2\ **Last updated:** [September 19, 2019, 4:26am UTC](https://discuss.elastic.co/t/how-can-i-write-to-logstash-without-filebeat-agent/200088 "2019-09-19T04:26:46Z")
Hi, Please let me know how can i write the messages to logstash without agent/filebeat. I would like to know if there are methods available to write into logstash without agent. Please help, Thanks [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=473) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=475)