# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=660 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 661 --- ## [Logstash & NSSM Windows - Variable for FQDN](https://discuss.elastic.co/t/logstash-nssm-windows-variable-for-fqdn/153181)
**Author:** [@londonx](https://discuss.elastic.co/u/londonx)\ **Replies:** 2\ **Last updated:** [October 25, 2018, 2:09pm UTC](https://discuss.elastic.co/t/logstash-nssm-windows-variable-for-fqdn/153181 "2018-10-25T14:09:46Z")
Has anyone managed to get the host FQDN ingested into Logstash when running it as a Windows Service via NSSM? filter { mutate { add\_field =\> { "HostFQDN" =\> "${COMPUTERNAME}.${USERDNSDOMAIN}" So this… --- ## [Why the mysql fields convert to lowercase](https://discuss.elastic.co/t/why-the-mysql-fields-convert-to-lowercase/152966)
**Author:** [@shinesllw](https://discuss.elastic.co/u/shinesllw)\ **Replies:** 2\ **Last updated:** [October 25, 2018, 12:34pm UTC](https://discuss.elastic.co/t/why-the-mysql-fields-convert-to-lowercase/152966 "2018-10-25T12:34:52Z")
I tried to use logstash to deliver the data from mysql to elasticsearch. but when i run the jdbc.conf, i found that all the fields in mysql were converted to lowsecase which exactly was not my hope. Can someone help me t… --- ## [Retrieve and set SNMP if Index name in logstash netflow codec out data](https://discuss.elastic.co/t/retrieve-and-set-snmp-if-index-name-in-logstash-netflow-codec-out-data/153996)
**Author:** [@lkpatel](https://discuss.elastic.co/u/lkpatel)\ **Replies:** 0\ **Last updated:** [October 25, 2018, 12:10pm UTC](https://discuss.elastic.co/t/retrieve-and-set-snmp-if-index-name-in-logstash-netflow-codec-out-data/153996 "2018-10-25T12:10:06Z")
Dear All, Is there any way to retrieve and set SNMP if index name in netflow out data ? Thanks and Regards Lalit Patel --- ## [Is it possible to change output structure](https://discuss.elastic.co/t/is-it-possible-to-change-output-structure/153759)
**Author:** [@humalog](https://discuss.elastic.co/u/humalog)\ **Replies:** 24\ **Last updated:** [October 25, 2018, 11:47am UTC](https://discuss.elastic.co/t/is-it-possible-to-change-output-structure/153759 "2018-10-25T11:47:49Z")
Hello, I got an output structure like that: { "path" =\> "/home/data/test.xml", "@version" =\> "1", "@timestamp" =\> 2018-10-24T08:49:15.480Z, "Properties" =\> { "tag" =\> … --- ## [How can I create index per hostname per day?](https://discuss.elastic.co/t/how-can-i-create-index-per-hostname-per-day/153992)
**Author:** [@09958a6c84c61fe52414](https://discuss.elastic.co/u/09958a6c84c61fe52414)\ **Replies:** 2\ **Last updated:** [October 25, 2018, 11:39am UTC](https://discuss.elastic.co/t/how-can-i-create-index-per-hostname-per-day/153992 "2018-10-25T11:39:50Z")
Hi all! Now I am using logstash config that makes every day new index for all incoming data (from many hosts) with name like "filebeat-2018.10.25" Config is: input { beats { port =\> 5044 } } filt… --- ## [Netflow Module (--setup) Broken](https://discuss.elastic.co/t/netflow-module-setup-broken/151652)
**Author:** [@Clay\_Curtis](https://discuss.elastic.co/u/Clay_Curtis)\ **Replies:** 1\ **Last updated:** [October 25, 2018, 11:31am UTC](https://discuss.elastic.co/t/netflow-module-setup-broken/151652 "2018-10-25T11:31:28Z")
I have been messing with the most basic thing for two days now and cannot get it to function. I am trying to run bin/logstash --modules netflow --setup and it fails. Without --setup it doesn't crash, but I need it to co… --- ## [Help with index patterns used in netflow](https://discuss.elastic.co/t/help-with-index-patterns-used-in-netflow/153081)
**Author:** [@gsantiago](https://discuss.elastic.co/u/gsantiago)\ **Replies:** 2\ **Last updated:** [October 25, 2018, 11:26am UTC](https://discuss.elastic.co/t/help-with-index-patterns-used-in-netflow/153081 "2018-10-25T11:26:06Z")
I've setup elastiflow on my ELK Stack to serve as a netflow collector, but I noticed visualizations showed different data to what I expected, then I realized kibana was using flow.client\_hostname index pattern as the fie… --- ## [Create new json from existing json](https://discuss.elastic.co/t/create-new-json-from-existing-json/153983)
**Author:** [@Zeeshan\_Alam](https://discuss.elastic.co/u/Zeeshan_Alam)\ **Replies:** 0\ **Last updated:** [October 25, 2018, 11:10am UTC](https://discuss.elastic.co/t/create-new-json-from-existing-json/153983 "2018-10-25T11:10:50Z")
I am getting data in this format: { "a": "1", "b": "2" } I want it to changes to: { data: { "a": "1", "b": "2" } } I can change it using the mutate filter: mutate { rename =\> { "\[a\]" =\> "\[data\]\[a\]" "\[b… --- ## [Receiving flume message](https://discuss.elastic.co/t/receiving-flume-message/153982)
**Author:** [@pjanzen](https://discuss.elastic.co/u/pjanzen)\ **Replies:** 0\ **Last updated:** [October 25, 2018, 11:10am UTC](https://discuss.elastic.co/t/receiving-flume-message/153982 "2018-10-25T11:10:46Z")
Hi All, I have a source that is determent to use Flume to send message to my ELK stack. I setup a tcp logstash input and I am recieving data, this data however is send in batches from flume. Is there any one who has don… --- ## [Logstash rpm install on custom location](https://discuss.elastic.co/t/logstash-rpm-install-on-custom-location/153968)
**Author:** [@chitraj8](https://discuss.elastic.co/u/chitraj8)\ **Replies:** 0\ **Last updated:** [October 25, 2018, 10:29am UTC](https://discuss.elastic.co/t/logstash-rpm-install-on-custom-location/153968 "2018-10-25T10:29:18Z")
Hi I am trying to understand how to provide custom path for logstash rpm in centos machine. We dont want to use the default root folders and thus would like to get the installation directory structure created on custom… --- ## [Logstash Cluster with two nodes](https://discuss.elastic.co/t/logstash-cluster-with-two-nodes/153965)
**Author:** [@Vijay\_Singh](https://discuss.elastic.co/u/Vijay_Singh)\ **Replies:** 1\ **Last updated:** [October 25, 2018, 10:29am UTC](https://discuss.elastic.co/t/logstash-cluster-with-two-nodes/153965 "2018-10-25T10:29:02Z")
Hi All, I am trying to run Logstash with 2 nodes in cluster with HA proxy setup. will this work. if yes than what configuration are required to achieve synchronization between the two nodes. --- ## [Logstash sizing for cores](https://discuss.elastic.co/t/logstash-sizing-for-cores/153957)
**Author:** [@mruthyu](https://discuss.elastic.co/u/mruthyu)\ **Replies:** 1\ **Last updated:** [October 25, 2018, 10:05am UTC](https://discuss.elastic.co/t/logstash-sizing-for-cores/153957 "2018-10-25T10:05:43Z")
We have a requirement to process 20000 events/second. Logstash uses few GROK filters for event processing. Would like to know how many cores and nodes are required. --- ## [Logstash index automatically delete when new is created](https://discuss.elastic.co/t/logstash-index-automatically-delete-when-new-is-created/153583)
**Author:** [@dinesh2](https://discuss.elastic.co/u/dinesh2)\ **Replies:** 3\ **Last updated:** [October 25, 2018, 9:00am UTC](https://discuss.elastic.co/t/logstash-index-automatically-delete-when-new-is-created/153583 "2018-10-25T09:00:50Z")
I want to create a script that previous day index got deleted when new index created in logstash config file. Index1: 2018-01-01 Index2: 2018-01-02 . . . IndexN: Regards, Dinesh D --- ## [Logstash Bulk Indexing](https://discuss.elastic.co/t/logstash-bulk-indexing/153924)
**Author:** [@Panda8](https://discuss.elastic.co/u/Panda8)\ **Replies:** 6\ **Last updated:** [October 25, 2018, 8:54am UTC](https://discuss.elastic.co/t/logstash-bulk-indexing/153924 "2018-10-25T08:54:50Z")
Hello Community, I am new to Logstash and I was looking through the Logstash documentation for support of Bulk Indexing in elasticsearch output plugin. I got one config "bulk\_path" related to this. But I am unable to un… --- ## [Extract fields from json file in Logstash using json filter](https://discuss.elastic.co/t/extract-fields-from-json-file-in-logstash-using-json-filter/153948)
**Author:** [@abhi\_latpate](https://discuss.elastic.co/u/abhi_latpate)\ **Replies:** 0\ **Last updated:** [October 25, 2018, 8:34am UTC](https://discuss.elastic.co/t/extract-fields-from-json-file-in-logstash-using-json-filter/153948 "2018-10-25T08:34:30Z")
I want to extract fields from json file and want to create graphs in kibana My input.json file consists of similar json elements (Pasting small portion since file is huge): "tags": \[ "jdkinstall", "clas… --- ## [Logstash JDBC Input table data not in sync with Kibana Index](https://discuss.elastic.co/t/logstash-jdbc-input-table-data-not-in-sync-with-kibana-index/153395)
**Author:** [@groverjatin17](https://discuss.elastic.co/u/groverjatin17)\ **Replies:** 8\ **Last updated:** [October 25, 2018, 7:48am UTC](https://discuss.elastic.co/t/logstash-jdbc-input-table-data-not-in-sync-with-kibana-index/153395 "2018-10-25T07:48:12Z")
Hello All, I want an create a Dashboard with Visualization based on data from an Oracle Database. Since i want it to be real time means if new row is inserted in the table the same to be reflected back in the index in … --- ## [Help parsing Xml File](https://discuss.elastic.co/t/help-parsing-xml-file/152538)
**Author:** [@L-Luciano](https://discuss.elastic.co/u/L-Luciano)\ **Replies:** 3\ **Last updated:** [October 25, 2018, 7:53am UTC](https://discuss.elastic.co/t/help-parsing-xml-file/152538 "2018-10-25T07:53:32Z")
Hello, I tried to format my xml in ES. Xml Format : \ \ \ **Author:** [@N\_Max](https://discuss.elastic.co/u/N_Max)\ **Replies:** 0\ **Last updated:** [October 25, 2018, 7:40am UTC](https://discuss.elastic.co/t/extract-the-messages-from-edifact/153932 "2018-10-25T07:40:08Z") I'm new to logstash and I have a log file which is having messages, each message starts and ends with specific terms,between those start and end terms will be couple lines information I want to get those messages. pleas… --- ## [Grok Parse Failure - CSV input via Filebeat](https://discuss.elastic.co/t/grok-parse-failure-csv-input-via-filebeat/153662)
**Author:** [@phoenix66](https://discuss.elastic.co/u/phoenix66)\ **Replies:** 2\ **Last updated:** [October 25, 2018, 7:24am UTC](https://discuss.elastic.co/t/grok-parse-failure-csv-input-via-filebeat/153662 "2018-10-25T07:24:53Z")
Hi all, I am having a problem with a Grok filter failing to parse, despite it working fine in the Kibana Grok debugger. Here is an example message (CSV into Filebeat to Logstash) { "message" =\> "\\"46642378\\",\\… --- ## [How can i parse the data with comma seperated in logstash](https://discuss.elastic.co/t/how-can-i-parse-the-data-with-comma-seperated-in-logstash/153880)
**Author:** [@rajkumar.m](https://discuss.elastic.co/u/rajkumar.m)\ **Replies:** 4\ **Last updated:** [October 25, 2018, 6:58am UTC](https://discuss.elastic.co/t/how-can-i-parse-the-data-with-comma-seperated-in-logstash/153880 "2018-10-25T06:58:23Z")
I have the below log format which is sending it to logstash through filebeat. I would like to create fields from these and send it to kibana. server1, cpu, 10, xinfo xyz wwer 2, xyzz server2, mem, 25 , xyzinfo abc 123… --- ## [How to delete index and schedule data in logstash config file](https://discuss.elastic.co/t/how-to-delete-index-and-schedule-data-in-logstash-config-file/153844)
**Author:** [@Nethaji](https://discuss.elastic.co/u/Nethaji)\ **Replies:** 5\ **Last updated:** [October 25, 2018, 6:55am UTC](https://discuss.elastic.co/t/how-to-delete-index-and-schedule-data-in-logstash-config-file/153844 "2018-10-25T06:55:17Z")
Hi I have scheduled my data in logstash config file it's working fine but the duplicate data will come in index. so i tried to delete the index first and then update the data but it's not working that is config file is … --- ## [Logstash stops reading from Kafka](https://discuss.elastic.co/t/logstash-stops-reading-from-kafka/153923)
**Author:** [@jibsonline](https://discuss.elastic.co/u/jibsonline)\ **Replies:** 0\ **Last updated:** [October 25, 2018, 6:50am UTC](https://discuss.elastic.co/t/logstash-stops-reading-from-kafka/153923 "2018-10-25T06:50:33Z")
ES:6.3.2 Logstash: 6.3.2-1 input{ kafka{ bootstrap\_servers =\> 'kafka-end-point:9092' topics =\> \["topic1", "topic2", "topic3", "topic4","topic5","topic6","topic7","topic8"\] codec =\> "json" … --- ## [Handle nested xml file with logstash and ruby script](https://discuss.elastic.co/t/handle-nested-xml-file-with-logstash-and-ruby-script/153603)
**Author:** [@humalog](https://discuss.elastic.co/u/humalog)\ **Replies:** 3\ **Last updated:** [October 25, 2018, 6:45am UTC](https://discuss.elastic.co/t/handle-nested-xml-file-with-logstash-and-ruby-script/153603 "2018-10-25T06:45:53Z")
Hello, I have an XML structure \ \ \ \Arena Standard\ \ \max\_simult\_tcp\_sessions\>\ **Author:** [@peruma.selvarasu](https://discuss.elastic.co/u/peruma.selvarasu)\ **Replies:** 0\ **Last updated:** [October 25, 2018, 3:35am UTC](https://discuss.elastic.co/t/concurrency-causing-ruby-exception-logstash-5-4-3/153899 "2018-10-25T03:35:26Z") We are getting the below error in logstash-5.4.3. I am not seeing any transaction failure due to this issue. Is there any suggestion to fix the problem?? \[2018-10-16T18:53:21,745\]\[ERROR\]\[logstash.filters.ruby \] Ruby… --- ## [How to get the geo\_point from database by using logstash](https://discuss.elastic.co/t/how-to-get-the-geo-point-from-database-by-using-logstash/153896)
**Author:** [@e25f05fd961f91e287cb](https://discuss.elastic.co/u/e25f05fd961f91e287cb)\ **Replies:** 0\ **Last updated:** [October 25, 2018, 2:57am UTC](https://discuss.elastic.co/t/how-to-get-the-geo-point-from-database-by-using-logstash/153896 "2018-10-25T02:57:55Z")
I have the longtitude and latitude in database ,and the type is float. I want to use kibana to draw coordinate Map, how can I get the right form with the data. filter { mutate { add\_field =\> { … --- ## [Parsing issues](https://discuss.elastic.co/t/parsing-issues/153895)
**Author:** [@rcbart](https://discuss.elastic.co/u/rcbart)\ **Replies:** 0\ **Last updated:** [October 25, 2018, 2:50am UTC](https://discuss.elastic.co/t/parsing-issues/153895 "2018-10-25T02:50:50Z")
I'm a newbie to ES so pardon the ignorance or if I am posting in the wrong place. Here is the problem I am facing. Ive set up ES, filebeats, logstash and kibana on my local WIndows 10 machine. I have a sample apac… --- ## [Note needed](https://discuss.elastic.co/t/note-needed/153227)
**Author:** [@xpsserver](https://discuss.elastic.co/u/xpsserver)\ **Replies:** 0\ **Last updated:** [October 19, 2018, 9:10pm UTC](https://discuss.elastic.co/t/note-needed/153227 "2018-10-19T21:10:14Z")
Welcome to Discuss the Elastic Stack — thanks for contributing! Does your reply improve the conversation in some way? Be kind to your fellow community members. Constructive criticism is welcome, but criticize ideas ,… --- ## [Logstash generates duplicates when adding fields](https://discuss.elastic.co/t/logstash-generates-duplicates-when-adding-fields/153887)
**Author:** [@ratman](https://discuss.elastic.co/u/ratman)\ **Replies:** 0\ **Last updated:** [October 25, 2018, 12:41am UTC](https://discuss.elastic.co/t/logstash-generates-duplicates-when-adding-fields/153887 "2018-10-25T00:41:09Z")
Hi! I don't know what's going on but no matter what Logstash configuration I am using but it keeps on duplicating (2-3-4 times) all the events that go through my pipeline... I noticed that it duplicates the fields I am … --- ## [Remove space from date string](https://discuss.elastic.co/t/remove-space-from-date-string/151671)
**Author:** [@CraigFoote](https://discuss.elastic.co/u/CraigFoote)\ **Replies:** 2\ **Last updated:** [October 24, 2018, 9:31pm UTC](https://discuss.elastic.co/t/remove-space-from-date-string/151671 "2018-10-24T21:31:56Z")
I have a field named 'datetime' that is a string with a space between the date and the time (ISO8601 with a space instead of a 'T'). I need the date portion to use in another field. I've tried this: mutate{ add\_field =\>… --- ## [Need to index .txt file to elastic search using logstash](https://discuss.elastic.co/t/need-to-index-txt-file-to-elastic-search-using-logstash/153842)
**Author:** [@Deepak\_Jaganath](https://discuss.elastic.co/u/Deepak_Jaganath)\ **Replies:** 1\ **Last updated:** [October 24, 2018, 8:54pm UTC](https://discuss.elastic.co/t/need-to-index-txt-file-to-elastic-search-using-logstash/153842 "2018-10-24T20:54:58Z")
My txt file location is https://s3.us-east-2.amazonaws.com/elasticbeanstalk-us-east-2-981790547314/Affiliations.txt So technically not an .sql file. Any help in terms of how to configure logstash to ingest / index into … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=659) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=661)