# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=969 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 970 --- ## [XML LOGSTASH configuration](https://discuss.elastic.co/t/xml-logstash-configuration/88290)
**Author:** [@Aylwinns\_Tan](https://discuss.elastic.co/u/Aylwinns_Tan)\ **Replies:** 2\ **Last updated:** [June 8, 2017, 5:15am UTC](https://discuss.elastic.co/t/xml-logstash-configuration/88290 "2017-06-08T05:15:06Z")
Hi all, Im trying to configure logstash for an XML log but i cant seem to get it right. This is my configuration: xml { source =\> "message" target =\> "Full.XML" xpath =\> … --- ## [Parsing filebeat -\> kafka -\> logstash](https://discuss.elastic.co/t/parsing-filebeat-kafka-logstash/85729)
**Author:** [@Dominic\_Prittie](https://discuss.elastic.co/u/Dominic_Prittie)\ **Replies:** 1\ **Last updated:** [May 16, 2017, 3:14pm UTC](https://discuss.elastic.co/t/parsing-filebeat-kafka-logstash/85729 "2017-05-16T15:14:33Z")
Hi, I am using the following versions of software: filebeat: 5.4.0 kafka: 0.10.0.1 logstash: 5.4.0 When I directly connect filebeat to logstash my log's fields such as host and source get parsed sensibly, eg: { … --- ## [5.4 logstash-input-websocket](https://discuss.elastic.co/t/5-4-logstash-input-websocket/88609)
**Author:** [@chimbo84](https://discuss.elastic.co/u/chimbo84)\ **Replies:** 1\ **Last updated:** [June 7, 2017, 6:48pm UTC](https://discuss.elastic.co/t/5-4-logstash-input-websocket/88609 "2017-06-07T18:48:35Z")
Reported this on the github repo but it doesn't seem to have any traction yet. Trying to install logstash-input-websocket and am getting the dreaded java error: root@enb:~# /usr/share/logstash/bin/logstash-plugin ins… --- ## [Custom logstash index name output to ES](https://discuss.elastic.co/t/custom-logstash-index-name-output-to-es/88608)
**Author:** [@Brandon\_Reeves](https://discuss.elastic.co/u/Brandon_Reeves)\ **Replies:** 1\ **Last updated:** [June 7, 2017, 6:37pm UTC](https://discuss.elastic.co/t/custom-logstash-index-name-output-to-es/88608 "2017-06-07T18:37:27Z")
We would like to create a custom index in ES using hostname, date, and a variable from a file if possible. Here is our current output output { elasticsearch { hosts =\> "127.0.0.1" index =\> "logstash-%{+YY… --- ## [Dateparsefailure](https://discuss.elastic.co/t/dateparsefailure/88592)
**Author:** [@chimbo84](https://discuss.elastic.co/u/chimbo84)\ **Replies:** 3\ **Last updated:** [June 7, 2017, 6:17pm UTC](https://discuss.elastic.co/t/dateparsefailure/88592 "2017-06-07T18:17:00Z")
I have a log line that reads something like the following (there is a space between the date and time, not a new line): 2017-06-07 10:15:42.406424+00:00,CSLUR,3,###########,###########,0xffffffff,310,026,61002,R,15,U,H … --- ## [Issue with Multiple Logstash Conf files](https://discuss.elastic.co/t/issue-with-multiple-logstash-conf-files/87762)
**Author:** [@Sameer\_Panicker](https://discuss.elastic.co/u/Sameer_Panicker)\ **Replies:** 13\ **Last updated:** [June 7, 2017, 6:16pm UTC](https://discuss.elastic.co/t/issue-with-multiple-logstash-conf-files/87762 "2017-06-07T18:16:29Z")
I am facing a problem when multiple logstash conf are used within a folder and trying to connect the same using filebeat. I had posted this issue under filebeat section and now I am asked to post it under logstash as wel… --- ## [Problem with coding Grok filter](https://discuss.elastic.co/t/problem-with-coding-grok-filter/88484)
**Author:** [@ibrahimsharaf](https://discuss.elastic.co/u/ibrahimsharaf)\ **Replies:** 4\ **Last updated:** [June 7, 2017, 1:12pm UTC](https://discuss.elastic.co/t/problem-with-coding-grok-filter/88484 "2017-06-07T13:12:54Z")
Hi there, I have this message 2017-05-01 08:09:19 \[scraper.py\] ERROR: Error downloading \ I want to write a grok filter to extract this part Error downloading \ **Author:** [@deeps123](https://discuss.elastic.co/u/deeps123)\ **Replies:** 0\ **Last updated:** [June 7, 2017, 11:29am UTC](https://discuss.elastic.co/t/logstash-multiline-pattern/88564 "2017-06-07T11:29:15Z") Hi, How to write pattern inside multiline codec to append the data till nth occurrence of the string. Like eg- File content is this: public class Constr1 { private int a; public Constr1 () { Constr1 Constr1 Cons… --- ## [Mapper\_parsing\_exception for timestamp field](https://discuss.elastic.co/t/mapper-parsing-exception-for-timestamp-field/87159)
**Author:** [@Dee](https://discuss.elastic.co/u/Dee)\ **Replies:** 6\ **Last updated:** [June 7, 2017, 11:27am UTC](https://discuss.elastic.co/t/mapper-parsing-exception-for-timestamp-field/87159 "2017-06-07T11:27:58Z")
Hi, For some reason logstash can't parse a field called timestamp in my json events in some environments (on other environments it works fine) This is the error I see in logstash logs: "error"=\>{"type"=\>"mapper\_parsing… --- ## [How to create my own document\_id in logstash?](https://discuss.elastic.co/t/how-to-create-my-own-document-id-in-logstash/1416)
**Author:** [@elastic\_paul](https://discuss.elastic.co/u/elastic_paul)\ **Replies:** 18\ **Last updated:** [June 7, 2017, 11:19am UTC](https://discuss.elastic.co/t/how-to-create-my-own-document-id-in-logstash/1416 "2017-06-07T11:19:05Z")
I would like to create my own document\_id to avoid duplication. I would like to make the document\_id as an MD5 hash of two fields; "ip" and "sha1\_fingerprint". eg; in pseudo code: md5\_hex( "ip" + " sha1\_fingerprin… --- ## [Logstash not starting with 'Cannot allocate memory' error](https://discuss.elastic.co/t/logstash-not-starting-with-cannot-allocate-memory-error/88558)
**Author:** [@aarishramesh](https://discuss.elastic.co/u/aarishramesh)\ **Replies:** 0\ **Last updated:** [June 7, 2017, 10:59am UTC](https://discuss.elastic.co/t/logstash-not-starting-with-cannot-allocate-memory-error/88558 "2017-06-07T10:59:47Z")
I have set up ELK in my aws t2.medium instance (4 GB RAM) & it was working fine until today morning when logstash failed. This was the error I encountered while trying to start logstash Java HotSpot(TM) 64-Bit Server V… --- ## [Filebeat and Logstash Installation](https://discuss.elastic.co/t/filebeat-and-logstash-installation/88359)
**Author:** [@richagautam](https://discuss.elastic.co/u/richagautam)\ **Replies:** 4\ **Last updated:** [June 7, 2017, 10:44am UTC](https://discuss.elastic.co/t/filebeat-and-logstash-installation/88359 "2017-06-07T10:44:57Z")
Hi all, We have to develop a solution in which we have to give more than 1000 log files of around 2 GB to Logstash for parsing the logs and giving the structured data. I am planning to use Filebeat to give the log… --- ## [How to aggregate zimbra logs?](https://discuss.elastic.co/t/how-to-aggregate-zimbra-logs/88374)
**Author:** [@mkalinowski](https://discuss.elastic.co/u/mkalinowski)\ **Replies:** 2\ **Last updated:** [June 7, 2017, 10:19am UTC](https://discuss.elastic.co/t/how-to-aggregate-zimbra-logs/88374 "2017-06-07T10:19:28Z")
Hello I want to aggregate zimbra logs from serveral servers. Here is an short example : 85C19C0506: removed A4936C0506: client=unknown\[10.10.10.10\]:55288 A4936C0506: message-id=20170515220002.A4936C0506@mta.zimbra.… --- ## [How logstash read input file?](https://discuss.elastic.co/t/how-logstash-read-input-file/88363)
**Author:** [@henrilabarre](https://discuss.elastic.co/u/henrilabarre)\ **Replies:** 9\ **Last updated:** [June 7, 2017, 10:18am UTC](https://discuss.elastic.co/t/how-logstash-read-input-file/88363 "2017-06-07T10:18:56Z")
I wonder if Logstash read the file (csv for example) line per line, and one line after the previous one? and if not how can I be sure logstash parse the file depending of the line number order.... I ask this because in… --- ## [Grok array of jsons](https://discuss.elastic.co/t/grok-array-of-jsons/87484)
**Author:** [@Kiranmai\_Reddy](https://discuss.elastic.co/u/Kiranmai_Reddy)\ **Replies:** 4\ **Last updated:** [June 7, 2017, 9:57am UTC](https://discuss.elastic.co/t/grok-array-of-jsons/87484 "2017-06-07T09:57:16Z")
how do I grok array of jsons eg: \[2017-05-29 02:17:18\] INFO – \[ActivityServiceRest:89\] – \[{“callId”:”kjwefkjweqkfb”},{“callId”:”nwekgwnkqgkqr”},{“callId”:”ohjiwnwbnwbrk”}\] I want to store all the callId's in a field ca… --- ## [How to parse Logline](https://discuss.elastic.co/t/how-to-parse-logline/87736)
**Author:** [@hispeed](https://discuss.elastic.co/u/hispeed)\ **Replies:** 4\ **Last updated:** [June 7, 2017, 7:37am UTC](https://discuss.elastic.co/t/how-to-parse-logline/87736 "2017-06-07T07:37:08Z")
Hi, I have the following line which I recieve in Kibana successfully. Connection ArchivUser: CIFS client \[ArchivUser\] from \[192.168.0.15(IP:192.168.0.15)\] accessed the shared folder \[Archiv Filme\]. So: "Connection"… --- ## [What are logstash decompression types input that are available?](https://discuss.elastic.co/t/what-are-logstash-decompression-types-input-that-are-available/88522)
**Author:** [@sahar\_q](https://discuss.elastic.co/u/sahar_q)\ **Replies:** 0\ **Last updated:** [June 7, 2017, 7:30am UTC](https://discuss.elastic.co/t/what-are-logstash-decompression-types-input-that-are-available/88522 "2017-06-07T07:30:06Z")
So i just found out logstash5.x does not support gzip as input, and the community plugin is broken for the 5.x versions so i'd like to know if logstash support getting compressed input of any kind, sending a compressed d… --- ## [Logstash setting output to elasticsearch](https://discuss.elastic.co/t/logstash-setting-output-to-elasticsearch/88508)
**Author:** [@Juneyoung\_Oh](https://discuss.elastic.co/u/Juneyoung_Oh)\ **Replies:** 3\ **Last updated:** [June 7, 2017, 7:15am UTC](https://discuss.elastic.co/t/logstash-setting-output-to-elasticsearch/88508 "2017-06-07T07:15:59Z")
I am using logstash to store and analyze logs of an web application. But some reason logstash keep failing to send data to elasticsearch. The error message says "cannot create pipeline". 13 \[2017-06-07T05:56:36,815\]… --- ## [Logstash not sending data to ES](https://discuss.elastic.co/t/logstash-not-sending-data-to-es/88020)
**Author:** [@keshav](https://discuss.elastic.co/u/keshav)\ **Replies:** 4\ **Last updated:** [June 7, 2017, 6:57am UTC](https://discuss.elastic.co/t/logstash-not-sending-data-to-es/88020 "2017-06-07T06:57:59Z")
I have a logstash config which reads nginx logs and sends data to elasticsearch. The log file contains two endpoints viz /index and /info. Logstash parses the /index endpoint and sends the data to ES but it doesn't do th… --- ## [Logstash date filter converting TIMESTAMP field to my local zone?](https://discuss.elastic.co/t/logstash-date-filter-converting-timestamp-field-to-my-local-zone/88260)
**Author:** [@Yaswanth](https://discuss.elastic.co/u/Yaswanth)\ **Replies:** 1\ **Last updated:** [June 7, 2017, 6:14am UTC](https://discuss.elastic.co/t/logstash-date-filter-converting-timestamp-field-to-my-local-zone/88260 "2017-06-07T06:14:30Z")
HI, I have logs in with following TIMESTAMP field in it and i am sending them to ES using logstash. \[2017-05-18 00:00:05,871\] \[2017-05-18 00:00:06,161\] \[2017-05-18 00:00:06,249\] \[2017-05-18 15:30:06,339\] Normally if… --- ## [Logstash cant load](https://discuss.elastic.co/t/logstash-cant-load/88270)
**Author:** [@talial](https://discuss.elastic.co/u/talial)\ **Replies:** 1\ **Last updated:** [June 7, 2017, 6:13am UTC](https://discuss.elastic.co/t/logstash-cant-load/88270 "2017-06-07T06:13:14Z")
Hi, we have a 5 nodes elasticsearch cluster running 5.2.0, 2 of the nodes are configured as client nodes (data=false, master=false), 3 nodes are elasticsearch cluster (data=true, master=true) I install logstash in on… --- ## [Setting up date plugin with Delhi timezone](https://discuss.elastic.co/t/setting-up-date-plugin-with-delhi-timezone/54626)
**Author:** [@noemie](https://discuss.elastic.co/u/noemie)\ **Replies:** 4\ **Last updated:** [June 7, 2017, 5:57am UTC](https://discuss.elastic.co/t/setting-up-date-plugin-with-delhi-timezone/54626 "2017-06-07T05:57:53Z")
Hello, I'm trying to set up the @timestamp based on a log date. The date in the log is in Delhi timezone but the one stored in the @timestamp (or another field created to store it) is in London timezone. Here is my l… --- ## [SystemDate or time to elasticsearch fields](https://discuss.elastic.co/t/systemdate-or-time-to-elasticsearch-fields/86872)
**Author:** [@Samatha](https://discuss.elastic.co/u/Samatha)\ **Replies:** 5\ **Last updated:** [June 7, 2017, 5:53am UTC](https://discuss.elastic.co/t/systemdate-or-time-to-elasticsearch-fields/86872 "2017-06-07T05:53:51Z")
We are trying to set current system date to one of the elastic field while inserting records using logstash script. mutate { add\_field =\> {"logdate" =\> "%{\[+YYYY-MM-dd\]}"} } output { elasticsearch{ hosts =\> \["… --- ## [Strftime is not working Logshtash ruby code](https://discuss.elastic.co/t/strftime-is-not-working-logshtash-ruby-code/88008)
**Author:** [@rpuserh](https://discuss.elastic.co/u/rpuserh)\ **Replies:** 3\ **Last updated:** [June 6, 2017, 11:19pm UTC](https://discuss.elastic.co/t/strftime-is-not-working-logshtash-ruby-code/88008 "2017-06-06T23:19:35Z")
Hi, Im running logstash 5.4 and i need new add\_field with sub date string im trying filter like this ruby { code =\> "event.set('read\_time' , event.get('@timestamp').strftime('%Y-%m'))" } And it … --- ## [Parsing netflow data](https://discuss.elastic.co/t/parsing-netflow-data/35334)
**Author:** [@Ian\_Hayes](https://discuss.elastic.co/u/Ian_Hayes)\ **Replies:** 2\ **Last updated:** [June 6, 2017, 10:28pm UTC](https://discuss.elastic.co/t/parsing-netflow-data/35334 "2017-06-06T22:28:56Z")
Hello all- I'm trying to have Logstash help out with some Netflow data. It receives data nicely and sends it to Elasticsearch with no problem, but I'm trying to add an extra field that contains the connection duration. … --- ## [Logstash again is failing to create monitoring event](https://discuss.elastic.co/t/logstash-again-is-failing-to-create-monitoring-event/85147)
**Author:** [@seriv](https://discuss.elastic.co/u/seriv)\ **Replies:** 13\ **Last updated:** [June 6, 2017, 10:21pm UTC](https://discuss.elastic.co/t/logstash-again-is-failing-to-create-monitoring-event/85147 "2017-06-06T22:21:52Z")
Hi, I have elastic-5.4.0 (elasticsearch, logstash, and kibana) all with x-pack, and trying to get the last piece working: getting some events from logstash. Started with --log.level debug, I see the following in logsta… --- ## [Getting \_csvparsefailure](https://discuss.elastic.co/t/getting--csvparsefailure/88116)
**Author:** [@Amit2015](https://discuss.elastic.co/u/Amit2015)\ **Replies:** 1\ **Last updated:** [June 6, 2017, 7:27pm UTC](https://discuss.elastic.co/t/getting--csvparsefailure/88116 "2017-06-06T19:27:13Z")
Hi, I am getting \_csvparsefailure error. How can I solve it and parse logs successfully into ES. Help me please. --- ## [Kafka logstash plugin getting failed connection logs](https://discuss.elastic.co/t/kafka-logstash-plugin-getting-failed-connection-logs/88465)
**Author:** [@Bruno\_Valli](https://discuss.elastic.co/u/Bruno_Valli)\ **Replies:** 0\ **Last updated:** [June 6, 2017, 7:21pm UTC](https://discuss.elastic.co/t/kafka-logstash-plugin-getting-failed-connection-logs/88465 "2017-06-06T19:21:30Z")
I am using the Kafka plugin for Logstash 5.4.1, my kafka instance is protected using a ssl keystore. I looked at the logs, but other than the configuration getting loaded I don't see any attempts of connection from lo… --- ## [Errors when Logstash is writing to InfluxDB](https://discuss.elastic.co/t/errors-when-logstash-is-writing-to-influxdb/87807)
**Author:** [@Steve\_1](https://discuss.elastic.co/u/Steve_1)\ **Replies:** 3\ **Last updated:** [June 6, 2017, 5:48pm UTC](https://discuss.elastic.co/t/errors-when-logstash-is-writing-to-influxdb/87807 "2017-06-06T17:48:51Z")
Hi, I am going to send logs from Logstash to both Elasticsearch and InfluxDB. This is logstash.conf: input { beats { port =\> 5044 ssl =\> false ssl\_certificate =\> "..." ssl\_key =\> "..." } } filter… --- ## [Logstash run time error export JAVA\_HOME=/usr/lib/jvm/jre-1.6.0/](https://discuss.elastic.co/t/logstash-run-time-error-export-java-home-usr-lib-jvm-jre-1-6-0/88337)
**Author:** [@filebeater](https://discuss.elastic.co/u/filebeater)\ **Replies:** 1\ **Last updated:** [June 6, 2017, 5:46pm UTC](https://discuss.elastic.co/t/logstash-run-time-error-export-java-home-usr-lib-jvm-jre-1-6-0/88337 "2017-06-06T17:46:49Z")
Hello, i have a newbie question. I think i have the correct java version, however, i got the following error from running logstash. any pointer is appreciated! echo $JAVA\_HOME /export/apps/jdk/JDK-1\_8\_0\_5 After I upda… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=968) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=970)