# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1229 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1230 --- ## [Kafka input json](https://discuss.elastic.co/t/kafka-input-json/32567)
**Author:** [@afreen](https://discuss.elastic.co/u/afreen)\ **Replies:** 1\ **Last updated:** [October 20, 2015, 11:58am UTC](https://discuss.elastic.co/t/kafka-input-json/32567 "2015-10-20T11:58:15Z")
Hi, I have input coming from kafka topic to logstash. Which is json data. JSON structure of my data "field1" : "val1", "field2" : "val2", "field3" : { "field4" : { "field5" : "val5", "field6" : "val6" } } input … --- ## [Forwarder only sending one logfile](https://discuss.elastic.co/t/forwarder-only-sending-one-logfile/32274)
**Author:** [@ssi](https://discuss.elastic.co/u/ssi)\ **Replies:** 4\ **Last updated:** [October 20, 2015, 10:12am UTC](https://discuss.elastic.co/t/forwarder-only-sending-one-logfile/32274 "2015-10-20T10:12:39Z")
i've added to log files to be shipped, if i do them individually no problem if i try and send both - only one of them end up in elasticsearch ? what gives? 015/10/15 15:11:41.183630 Loading registrar data from /opt/log… --- ## [Logstash filter ruby create new event?](https://discuss.elastic.co/t/logstash-filter-ruby-create-new-event/27377)
**Author:** [@quejinho](https://discuss.elastic.co/u/quejinho)\ **Replies:** 5\ **Last updated:** [October 20, 2015, 7:43am UTC](https://discuss.elastic.co/t/logstash-filter-ruby-create-new-event/27377 "2015-10-20T07:43:34Z")
I am planning to create a time difference betwen one logstash and another so in second one i made a ruby code to make a time difference. Although i am quite there i need to not add a field to an event but to create a ne… --- ## [Logstash Forwarder on 2 machines](https://discuss.elastic.co/t/logstash-forwarder-on-2-machines/32545)
**Author:** [@tatushar3](https://discuss.elastic.co/u/tatushar3)\ **Replies:** 3\ **Last updated:** [October 20, 2015, 6:46am UTC](https://discuss.elastic.co/t/logstash-forwarder-on-2-machines/32545 "2015-10-20T06:46:32Z")
ogstash forwarder is installed on two machines and both are shipping logs to a single machine where logstash is installed.How do I distinguish between the logs of two machines --- ## [Ctrl A separated csv file elasticsearch not indexing using logstash](https://discuss.elastic.co/t/ctrl-a-separated-csv-file-elasticsearch-not-indexing-using-logstash/32077)
**Author:** [@Avinash\_Garg](https://discuss.elastic.co/u/Avinash_Garg)\ **Replies:** 4\ **Last updated:** [October 20, 2015, 6:44am UTC](https://discuss.elastic.co/t/ctrl-a-separated-csv-file-elasticsearch-not-indexing-using-logstash/32077 "2015-10-20T06:44:04Z")
0 down vote favorite I am using the config: input { file { path =\> "/tmp/test12.csv" type =\> "Sample" start\_position =\> "beginning" } } filter { csv { columns =\> … --- ## [How to print the logs of logstash execution?](https://discuss.elastic.co/t/how-to-print-the-logs-of-logstash-execution/32539)
**Author:** [@Gaurav\_Varma](https://discuss.elastic.co/u/Gaurav_Varma)\ **Replies:** 1\ **Last updated:** [October 20, 2015, 12:09am UTC](https://discuss.elastic.co/t/how-to-print-the-logs-of-logstash-execution/32539 "2015-10-20T00:09:22Z")
Is there a way we can print logstash execution output to a log file? For example, I am using a jdbc plugin to read data as per sql\_last\_start. I want to know at what time the query was executed, how many records in re… --- ## [Can I set \_ttl in a Logstash config file?](https://discuss.elastic.co/t/can-i-set--ttl-in-a-logstash-config-file/32308)
**Author:** [@CraigFoote](https://discuss.elastic.co/u/CraigFoote)\ **Replies:** 2\ **Last updated:** [October 19, 2015, 5:01pm UTC](https://discuss.elastic.co/t/can-i-set--ttl-in-a-logstash-config-file/32308 "2015-10-19T17:01:20Z")
I see lots of documentation on setting the TTL in a template file to be applied to all indices but can I set them individually? I tried using add\_field for ttl but got a configtest error saying ""-prefixed fields cannot … --- ## [Is there a way to create nested structures in logstash?](https://discuss.elastic.co/t/is-there-a-way-to-create-nested-structures-in-logstash/32509)
**Author:** [@dlang](https://discuss.elastic.co/u/dlang)\ **Replies:** 1\ **Last updated:** [October 19, 2015, 4:46pm UTC](https://discuss.elastic.co/t/is-there-a-way-to-create-nested-structures-in-logstash/32509 "2015-10-19T16:46:26Z")
For example, I want to have the output be something like: { "trusted" : { "logstash": { "host": "some IP", "timereceived": "some timestamp" } } } does logstash follow the project lumberjac… --- ## [Possible values for "fields type" variable in a logstash forwarder config?](https://discuss.elastic.co/t/possible-values-for-fields-type-variable-in-a-logstash-forwarder-config/32503)
**Author:** [@xtingray](https://discuss.elastic.co/u/xtingray)\ **Replies:** 1\ **Last updated:** [October 19, 2015, 4:06pm UTC](https://discuss.elastic.co/t/possible-values-for-fields-type-variable-in-a-logstash-forwarder-config/32503 "2015-10-19T16:06:06Z")
Hi, I am configuring a logstash forwarder where there is a log created by my own application. This log has no relation to any syntax I have seen in several examples (syslog, apache, etc). So, when one is working with a v… --- ## [Mutate and empty values](https://discuss.elastic.co/t/mutate-and-empty-values/32498)
**Author:** [@maisyweb](https://discuss.elastic.co/u/maisyweb)\ **Replies:** 2\ **Last updated:** [October 19, 2015, 3:18pm UTC](https://discuss.elastic.co/t/mutate-and-empty-values/32498 "2015-10-19T15:18:43Z")
Hi, I'm using the mutate plugin to remove square brackets i'm getting while trying to transform an XML file into a json file mutate { replace =\> \["checkInFrom", "%{\[checkInFrom\]\[0\]}"\] } This is working well. The on… --- ## [Error while starting up logstash block device detection unsupported or native support failed to load](https://discuss.elastic.co/t/error-while-starting-up-logstash-block-device-detection-unsupported-or-native-support-failed-to-load/32090)
**Author:** [@Shailen](https://discuss.elastic.co/u/Shailen)\ **Replies:** 3\ **Last updated:** [October 19, 2015, 2:23pm UTC](https://discuss.elastic.co/t/error-while-starting-up-logstash-block-device-detection-unsupported-or-native-support-failed-to-load/32090 "2015-10-19T14:23:16Z")
I have been running logstash 1.5.4, elasticsearch 1.7.2 and kibana 4.1.2 for a couple of weeks without any issues. The environment is as follows: $ uname -a Linux picdsr000001076 2.6.32-504.16.2.el6.x86\_64 #1 SMP Tue M… --- ## [Logstash with Kafka: Unable to decode avro](https://discuss.elastic.co/t/logstash-with-kafka-unable-to-decode-avro/32457)
**Author:** [@mgarg](https://discuss.elastic.co/u/mgarg)\ **Replies:** 1\ **Last updated:** [October 19, 2015, 10:18am UTC](https://discuss.elastic.co/t/logstash-with-kafka-unable-to-decode-avro/32457 "2015-10-19T10:18:13Z")
I am trying to consume serialized avro events from a Kafka Queue. The kafka queue is populated using a simple java producer. For clarity I am sharing the three components: Avro schema file {"namespace": "example.avro… --- ## [LSF excluding binaries or specific file types](https://discuss.elastic.co/t/lsf-excluding-binaries-or-specific-file-types/32404)
**Author:** [@ktella](https://discuss.elastic.co/u/ktella)\ **Replies:** 1\ **Last updated:** [October 19, 2015, 6:16am UTC](https://discuss.elastic.co/t/lsf-excluding-binaries-or-specific-file-types/32404 "2015-10-19T06:16:28Z")
Friends, I have a log file in my App server as below and LSF is shipping well. /opt/jboss-eap/server/xxxxxxxxxx/log/localhost\_access\_log.2015-10-16.log We have a log archiving script which archives the old files with… --- ## [Elapsed plugin error](https://discuss.elastic.co/t/elapsed-plugin-error/32453)
**Author:** [@Anmol\_Gupta](https://discuss.elastic.co/u/Anmol_Gupta)\ **Replies:** 0\ **Last updated:** [October 19, 2015, 6:01am UTC](https://discuss.elastic.co/t/elapsed-plugin-error/32453 "2015-10-19T06:01:01Z")
i have a log file which looks like this: 2015-10-19 11:15:11 TRACE message start 2015-10-19 11:15:11 INFO booting complete! 2015-10-19 11:15:11 INFO booting complete! 2015-10-19 11:15:11 INFO booting complete! 2015-10-19… --- ## [How to stop logstash send data to mongo if data incorrect](https://discuss.elastic.co/t/how-to-stop-logstash-send-data-to-mongo-if-data-incorrect/32242)
**Author:** [@Jason\_Zheng](https://discuss.elastic.co/u/Jason_Zheng)\ **Replies:** 5\ **Last updated:** [October 19, 2015, 2:36am UTC](https://discuss.elastic.co/t/how-to-stop-logstash-send-data-to-mongo-if-data-incorrect/32242 "2015-10-19T02:36:09Z")
Hi All, It seems to a loop which cannot stop when logstash would like to write incorrect data to mongo, even I did 'service logstash restart', the logstash started and then start to try send the incorrect data My data … --- ## [Logstash JDBC Plugin - optimal way to fetch new incoming data](https://discuss.elastic.co/t/logstash-jdbc-plugin-optimal-way-to-fetch-new-incoming-data/32441)
**Author:** [@samnik60](https://discuss.elastic.co/u/samnik60)\ **Replies:** 0\ **Last updated:** [October 18, 2015, 7:48am UTC](https://discuss.elastic.co/t/logstash-jdbc-plugin-optimal-way-to-fetch-new-incoming-data/32441 "2015-10-18T07:48:18Z")
Hi guys, Aim : To take data from a oracle table at regular interval without overlapping data ( i.e data from first run doesnt show up in next run ) to a target . problem : How to query the table for the above use cas… --- ## [How to send log that has many line breaks as one entry](https://discuss.elastic.co/t/how-to-send-log-that-has-many-line-breaks-as-one-entry/31967)
**Author:** [@M.Tyler](https://discuss.elastic.co/u/M.Tyler)\ **Replies:** 4\ **Last updated:** [October 17, 2015, 11:08am UTC](https://discuss.elastic.co/t/how-to-send-log-that-has-many-line-breaks-as-one-entry/31967 "2015-10-17T11:08:40Z")
Please let me question how to collect Windows Defender log using logstash. Windows Defender output log such as the below sample when virus is detected. But the log has a lot of a line breaks in one detection log. So lo… --- ## [(Website) Migration of logstash.net to elastic.co](https://discuss.elastic.co/t/website-migration-of-logstash-net-to-elastic-co/32418)
**Author:** [@jordansissel](https://discuss.elastic.co/u/jordansissel)\ **Replies:** 6\ **Last updated:** [October 22, 2015, 6:33pm UTC](https://discuss.elastic.co/t/website-migration-of-logstash-net-to-elastic-co/32418 "2015-10-22T18:33:37Z")
Hello folks! It's been a wonderful many years that logstash.net has served us documentation, hasn't it? In our efforts to improve our documentation, we also migrated most of our newer Logstash docs to elastic.co. That … --- ## [Multiple patterns for Multiline Configuration?](https://discuss.elastic.co/t/multiple-patterns-for-multiline-configuration/32258)
**Author:** [@TimTim](https://discuss.elastic.co/u/TimTim)\ **Replies:** 2\ **Last updated:** [October 16, 2015, 7:06am UTC](https://discuss.elastic.co/t/multiple-patterns-for-multiline-configuration/32258 "2015-10-16T07:06:49Z")
I am wondering if it is possible to have mulitple patterns in the multiline configuration or Logstash (running 1.4.5 at the moment)? The reason is that we have several log files coming in that we want to go to the same… --- ## [High CPU usage for logstash-agent](https://discuss.elastic.co/t/high-cpu-usage-for-logstash-agent/32325)
**Author:** [@joiceedattel](https://discuss.elastic.co/u/joiceedattel)\ **Replies:** 1\ **Last updated:** [October 16, 2015, 3:40am UTC](https://discuss.elastic.co/t/high-cpu-usage-for-logstash-agent/32325 "2015-10-16T03:40:10Z")
The logstash-agent is consuming more than 100% of CPU in our nodes. Is there any solution to fix this issue . The logstash-agent.conf is as follows input { file { type =\> "hybris\_console" … --- ## [Encoding in logstash configure file and there must be space before end of Exec plugin argument](https://discuss.elastic.co/t/encoding-in-logstash-configure-file-and-there-must-be-space-before-end-of-exec-plugin-argument/32228)
**Author:** [@ShinJJang](https://discuss.elastic.co/u/ShinJJang)\ **Replies:** 7\ **Last updated:** [October 16, 2015, 1:49am UTC](https://discuss.elastic.co/t/encoding-in-logstash-configure-file-and-there-must-be-space-before-end-of-exec-plugin-argument/32228 "2015-10-16T01:49:28Z")
Hi there, I have problem using korean(utf-8) arguments in exec output. output { command =\> "test.sh '' '{한글과 English}' 'somthing' " # If there is not end space '(single quote) with "(double quote), error occ… --- ## [Logstash fails get logs from UDP](https://discuss.elastic.co/t/logstash-fails-get-logs-from-udp/32105)
**Author:** [@thomas.dotreppe](https://discuss.elastic.co/u/thomas.dotreppe)\ **Replies:** 1\ **Last updated:** [October 15, 2015, 10:48pm UTC](https://discuss.elastic.co/t/logstash-fails-get-logs-from-udp/32105 "2015-10-15T22:48:25Z")
Here is the test configuration I'm using to debug the issue: input { udp { workers =\> 5 type =\> "fortinet" port =\> 1514 queue\_size =\> 100000 … --- ## [Need help with configuration](https://discuss.elastic.co/t/need-help-with-configuration/32166)
**Author:** [@m\_abdelfattah](https://discuss.elastic.co/u/m_abdelfattah)\ **Replies:** 6\ **Last updated:** [October 15, 2015, 8:10pm UTC](https://discuss.elastic.co/t/need-help-with-configuration/32166 "2015-10-15T20:10:25Z")
I have the following setup for an ELK Platform: logstash-forwarder on 20 machines as following: 4 machines (Nginx + sys logs) 4 machines (PostgreSQL) 12 machines (Ruby on Rails + sys logs) And I have one machine tha… --- ## [Basic Configuration File](https://discuss.elastic.co/t/basic-configuration-file/32215)
**Author:** [@Didier](https://discuss.elastic.co/u/Didier)\ **Replies:** 4\ **Last updated:** [October 15, 2015, 8:09pm UTC](https://discuss.elastic.co/t/basic-configuration-file/32215 "2015-10-15T20:09:54Z")
Hello, I have the following code as a test configuration file. input { file { path =\> "C:\\Users\\Admin\\Desktop\\Logs\_elasticsearch\\test\_10142015.txt" } } filter { grok { patterns\_dir =\> "C:\\Program Files\\lo… --- ## [Parsing a crappy date format](https://discuss.elastic.co/t/parsing-a-crappy-date-format/32263)
**Author:** [@Aviv\_Ratzon](https://discuss.elastic.co/u/Aviv_Ratzon)\ **Replies:** 10\ **Last updated:** [October 15, 2015, 7:03pm UTC](https://discuss.elastic.co/t/parsing-a-crappy-date-format/32263 "2015-10-15T19:03:34Z")
Hi everyone, I have a real problem that I couldn't find a decent solution to. I need to parse log files and extract dates inside these files. the date format is really crappy and looks like this: 10/09/15 13:55:57:20… --- ## [Can logstash output an XML file?](https://discuss.elastic.co/t/can-logstash-output-an-xml-file/32300)
**Author:** [@piyush.m82](https://discuss.elastic.co/u/piyush.m82)\ **Replies:** 1\ **Last updated:** [October 15, 2015, 5:47pm UTC](https://discuss.elastic.co/t/can-logstash-output-an-xml-file/32300 "2015-10-15T17:47:51Z")
Hello Guys, I am fairly new to logstash and want to know few things before implementing anything further. I have a requirement to write into logstash, logs which are in json format and then logstash should output an xm… --- ## [Best approach for loading complex xml](https://discuss.elastic.co/t/best-approach-for-loading-complex-xml/32297)
**Author:** [@FrankC](https://discuss.elastic.co/u/FrankC)\ **Replies:** 0\ **Last updated:** [October 15, 2015, 3:59pm UTC](https://discuss.elastic.co/t/best-approach-for-loading-complex-xml/32297 "2015-10-15T15:59:40Z")
Hi, I am still fairly new to the ELK stack. I have a project coming up that I would like to experiment loading the xml files into ES using LS. The xml have many nested objects. So far I have loaded fairly simple x… --- ## [Tags removed in 2.0.0.beta2?](https://discuss.elastic.co/t/tags-removed-in-2-0-0-beta2/32281)
**Author:** [@CraigFoote](https://discuss.elastic.co/u/CraigFoote)\ **Replies:** 2\ **Last updated:** [October 15, 2015, 2:29pm UTC](https://discuss.elastic.co/t/tags-removed-in-2-0-0-beta2/32281 "2015-10-15T14:29:03Z")
Am I reading this right, Logstash 2.0.0 beta2 released | Elastic Blog? Until the introduction of conditionals, the only way to selectively apply filters and outputs to some events was to set the type, tags and exclude\_… --- ## [How to cut and delete elements in a array with Logstash](https://discuss.elastic.co/t/how-to-cut-and-delete-elements-in-a-array-with-logstash/32278)
**Author:** [@arok](https://discuss.elastic.co/u/arok)\ **Replies:** 5\ **Last updated:** [October 15, 2015, 2:22pm UTC](https://discuss.elastic.co/t/how-to-cut-and-delete-elements-in-a-array-with-logstash/32278 "2015-10-15T14:22:38Z")
Hi everybody!! I have Json logs in a array as this: e":\[{"n":"3/0/1","st":"CONTENT","v":"Sensortag"}, {"n":"3/0/3","st":"CONTENT","v":"Contiki-develop-20150508-409-g2147b9e"}, {"n":"3/0/13","st":"CONTENT","v"… --- ## [Unable to insert data to OpenTSDB](https://discuss.elastic.co/t/unable-to-insert-data-to-opentsdb/32267)
**Author:** [@sanlop](https://discuss.elastic.co/u/sanlop)\ **Replies:** 0\ **Last updated:** [October 15, 2015, 12:36pm UTC](https://discuss.elastic.co/t/unable-to-insert-data-to-opentsdb/32267 "2015-10-15T12:36:33Z")
Hello all: I am trying to insert data from RabbitMQ to OpenTSDB. I can not see why is not inserting the data: Metric opc\_ignition.test has been created manually , but I have also set the option to create automatically. … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1228) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1230)