# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=931 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 932 --- ## [Nested field not getting processed by ingest processor plugin](https://discuss.elastic.co/t/nested-field-not-getting-processed-by-ingest-processor-plugin/96136)
**Author:** [@him117](https://discuss.elastic.co/u/him117)\ **Replies:** 0\ **Last updated:** [August 7, 2017, 4:31pm UTC](https://discuss.elastic.co/t/nested-field-not-getting-processed-by-ingest-processor-plugin/96136 "2017-08-07T16:31:49Z")
I am getting the below error from logstash, when trying to insert a document - ERROR -\> 14:58:11.352 \[\[main\]\>worker0\] WARN logstash.outputs.elasticsearch - Failed action. {:status=\>500, :action=\>\[“index”, {:\_id=\>“TKT2… --- ## [Grok pattern for java exception](https://discuss.elastic.co/t/grok-pattern-for-java-exception/95949)
**Author:** [@SandhyaRani](https://discuss.elastic.co/u/SandhyaRani)\ **Replies:** 2\ **Last updated:** [August 7, 2017, 3:33pm UTC](https://discuss.elastic.co/t/grok-pattern-for-java-exception/95949 "2017-08-07T15:33:48Z")
Hi there! I wanted to write grok match pattern for java exception 2016-12-16 21:28:05,668 ERROR \[int-http-28\] \[nbiws::::\] c.t.d.s.impl.DiagnosticServiceImpl - Error running a diagnostic workflow : 9003: Invalid argument… --- ## [Wrong Syntax](https://discuss.elastic.co/t/wrong-syntax/96118)
**Author:** [@ssasporta](https://discuss.elastic.co/u/ssasporta)\ **Replies:** 1\ **Last updated:** [August 7, 2017, 2:31pm UTC](https://discuss.elastic.co/t/wrong-syntax/96118 "2017-08-07T14:31:15Z")
My Filter in Logstash looks like that: } else if "issuer" in \[jsonstring\] { mutate { add\_field =\> \[ "issuer", "%{\[jsonstring\]\[issuer\]}" \] } I am getting the following in Kibana: W… --- ## [Unable to dissect logs based on a string pattern](https://discuss.elastic.co/t/unable-to-dissect-logs-based-on-a-string-pattern/96120)
**Author:** [@prydeep](https://discuss.elastic.co/u/prydeep)\ **Replies:** 0\ **Last updated:** [August 7, 2017, 2:01pm UTC](https://discuss.elastic.co/t/unable-to-dissect-logs-based-on-a-string-pattern/96120 "2017-08-07T14:01:41Z")
Hello, this is the filter I am using filter { if ((\[message\] =~ /Interface/)) { dissect { mapping =\> { “message” =\> “%{syslog\_timestamp} %{+syslog\_timestamp},%{} %{} %{device\_type} %{hostname}-%{h1}/%{} %{kernel\_lo… --- ## [UTF-8 in Windows](https://discuss.elastic.co/t/utf-8-in-windows/96092)
**Author:** [@Harvii\_Dent](https://discuss.elastic.co/u/Harvii_Dent)\ **Replies:** 0\ **Last updated:** [August 7, 2017, 11:48am UTC](https://discuss.elastic.co/t/utf-8-in-windows/96092 "2017-08-07T11:48:59Z")
Hello, There seems to be a problem with how UTF-8 input is handled in Logstash (5.5.0) on Windows (2008 R2/2012 R2); using the below config, any arabic input comes out as question marks ??????????, and this seems indepe… --- ## [Logstash fail to start](https://discuss.elastic.co/t/logstash-fail-to-start/96102)
**Author:** [@zozo6015](https://discuss.elastic.co/u/zozo6015)\ **Replies:** 3\ **Last updated:** [August 7, 2017, 1:52pm UTC](https://discuss.elastic.co/t/logstash-fail-to-start/96102 "2017-08-07T13:52:46Z")
Hello, I have a brand new logstash installation which is failing to start with the following error: \[2017-08-07T12:18:11,073\]\[ERROR\]\[logstash.pipeline \] Logstash failed to create queue {"exception"=\>"integer 2147… --- ## [Data movement is taking place in logstash but we are getting the error](https://discuss.elastic.co/t/data-movement-is-taking-place-in-logstash-but-we-are-getting-the-error/96115)
**Author:** [@JAYALAKSHMI](https://discuss.elastic.co/u/JAYALAKSHMI)\ **Replies:** 0\ **Last updated:** [August 7, 2017, 1:43pm UTC](https://discuss.elastic.co/t/data-movement-is-taking-place-in-logstash-but-we-are-getting-the-error/96115 "2017-08-07T13:43:11Z")
Hi All, We have configured the logstash as below. input { elasticsearch { hosts =\> "localhost:9202" index =\> "index\_name" size =\> 100 scroll =\> "1m" docinfo =\> true } } filter { mutate { remove\_field =\> \[ “@… --- ## [java.lang.OutOfMemoryError: Java heap space while indexing xml file](https://discuss.elastic.co/t/java-lang-outofmemoryerror-java-heap-space-while-indexing-xml-file/96082)
**Author:** [@aashishchauhan06](https://discuss.elastic.co/u/aashishchauhan06)\ **Replies:** 2\ **Last updated:** [August 7, 2017, 12:49pm UTC](https://discuss.elastic.co/t/java-lang-outofmemoryerror-java-heap-space-while-indexing-xml-file/96082 "2017-08-07T12:49:12Z")
HI, I’m indexing a XML file and given ElasticSearch as output but after approx. 2 hours i got OOM error like this java.lang.OutOfMemoryError: Java heap space Dumping heap to java\_pid6940.hprof ... Heap dump file create… --- ## [Translate filter with more than one input field](https://discuss.elastic.co/t/translate-filter-with-more-than-one-input-field/96104)
**Author:** [@AHurtado](https://discuss.elastic.co/u/AHurtado)\ **Replies:** 0\ **Last updated:** [August 7, 2017, 12:30pm UTC](https://discuss.elastic.co/t/translate-filter-with-more-than-one-input-field/96104 "2017-08-07T12:30:40Z")
Hi, Related to another question I uploaded, and related to translate filters, is it possible to use translate filters with more than one field as input? For example, I have this example: translate { dictionary\_path =\>… --- ## [Logstash implement error](https://discuss.elastic.co/t/logstash-implement-error/96088)
**Author:** [@hadivarp](https://discuss.elastic.co/u/hadivarp)\ **Replies:** 2\ **Last updated:** [August 7, 2017, 11:47am UTC](https://discuss.elastic.co/t/logstash-implement-error/96088 "2017-08-07T11:47:27Z")
I want to Implement logstash and follow every steps witch logstash document explain but Logstash gave me this error is any one know what should I do ? indent preformatted text by 4 spaces Caused by: java.lang.Illegal… --- ## [Read MongoDB data into logstash](https://discuss.elastic.co/t/read-mongodb-data-into-logstash/96089)
**Author:** [@Vivek\_Sharma](https://discuss.elastic.co/u/Vivek_Sharma)\ **Replies:** 0\ **Last updated:** [August 7, 2017, 11:24am UTC](https://discuss.elastic.co/t/read-mongodb-data-into-logstash/96089 "2017-08-07T11:24:29Z")
Hi I am trying to read the data from MongoDB into my logstash but I am getting error with driver file it seems. Below is the snippet of my logstash conf file: input { jdbc { jdbc\_driver\_library =\> "C:\\ELK\\logstash\\b… --- ## [How to use geoip logstash filter?](https://discuss.elastic.co/t/how-to-use-geoip-logstash-filter/93513)
**Author:** [@DharaniKumar](https://discuss.elastic.co/u/DharaniKumar)\ **Replies:** 36\ **Last updated:** [August 7, 2017, 11:23am UTC](https://discuss.elastic.co/t/how-to-use-geoip-logstash-filter/93513 "2017-08-07T11:23:02Z")
In my log file i've lat and lon values in separate fields. I cant able to use tile map in kibana. I have these details in my log file. Can some one guide me how to use it in to tilemap in kibana. Thanks in advance. --- ## [How to parse multiple json files with nested json arrays in it](https://discuss.elastic.co/t/how-to-parse-multiple-json-files-with-nested-json-arrays-in-it/96084)
**Author:** [@niravhjoshi](https://discuss.elastic.co/u/niravhjoshi)\ **Replies:** 0\ **Last updated:** [August 7, 2017, 10:36am UTC](https://discuss.elastic.co/t/how-to-parse-multiple-json-files-with-nested-json-arrays-in-it/96084 "2017-08-07T10:36:01Z")
Hi there, I am trying to parse bunch of json files using logstash. Following is my logstash conf file input { file {codec =\> multiline { pattern =\> '^\\{' negate =\> true… --- ## [Udp output fails with big json msg](https://discuss.elastic.co/t/udp-output-fails-with-big-json-msg/96083)
**Author:** [@michaell](https://discuss.elastic.co/u/michaell)\ **Replies:** 0\ **Last updated:** [August 7, 2017, 10:27am UTC](https://discuss.elastic.co/t/udp-output-fails-with-big-json-msg/96083 "2017-08-07T10:27:50Z")
i am using the udp output plugin to send json messages recieved from http and rabbitmq input. logstash version 5.5.1 everything works fine for small messages. when i hit the 32k json size - logstash crashes (the logst… --- ## [HTTP output module, JSON array input -\> JSON array output](https://discuss.elastic.co/t/http-output-module-json-array-input-json-array-output/96074)
**Author:** [@qbadx](https://discuss.elastic.co/u/qbadx)\ **Replies:** 0\ **Last updated:** [August 7, 2017, 9:31am UTC](https://discuss.elastic.co/t/http-output-module-json-array-input-json-array-output/96074 "2017-08-07T09:31:49Z")
Hello, I’m using the HTTP output module to send data to an API . The data in the input file are stored in a JSON array like : \[ {“a”:“data”}, {“b”:“data”} \] but the output is JSON lines like : {“a”:“data”} {“b”:“… --- ## [How to download aws cloudwatch log data?](https://discuss.elastic.co/t/how-to-download-aws-cloudwatch-log-data/95158)
**Author:** [@wolfghost](https://discuss.elastic.co/u/wolfghost)\ **Replies:** 6\ **Last updated:** [July 31, 2017, 10:27am UTC](https://discuss.elastic.co/t/how-to-download-aws-cloudwatch-log-data/95158 "2017-07-31T10:27:00Z")
Please tell me about How to download aws cloudwatch log data? --- ## [Pattern not defined even though the first matches](https://discuss.elastic.co/t/pattern-not-defined-even-though-the-first-matches/94929)
**Author:** [@Hisushi](https://discuss.elastic.co/u/Hisushi)\ **Replies:** 1\ **Last updated:** [August 7, 2017, 7:32am UTC](https://discuss.elastic.co/t/pattern-not-defined-even-though-the-first-matches/94929 "2017-08-07T07:32:03Z")
Hi everybody, I’m pretty new to the ELK stack but I think I understand the basics. So far, I’ve been able to define a pattern to match my log file and an additional grok pattern. I also checked if it works as expected … --- ## [Telnet 192.168.112.199 9200 connection refused](https://discuss.elastic.co/t/telnet-192-168-112-199-9200-connection-refused/95900)
**Author:** [@Joe\_Yang](https://discuss.elastic.co/u/Joe_Yang)\ **Replies:** 4\ **Last updated:** [August 7, 2017, 5:26am UTC](https://discuss.elastic.co/t/telnet-192-168-112-199-9200-connection-refused/95900 "2017-08-07T05:26:12Z")
we’d like to collect IIS log on windows windows, logstash and elasticesearch both installed on Linux(192.168.112.199), the following command failed on windows: telnet 192.168.112.199 9200 -\> connection refused ./fileb… --- ## [InfluxDB Output: parametric data\_points](https://discuss.elastic.co/t/influxdb-output-parametric-data-points/95938)
**Author:** [@JoaquinS](https://discuss.elastic.co/u/JoaquinS)\ **Replies:** 1\ **Last updated:** [August 7, 2017, 5:30am UTC](https://discuss.elastic.co/t/influxdb-output-parametric-data-points/95938 "2017-08-07T05:30:18Z")
Hello, I want to write this data into InfluxDB: { "product" =\> "telegraf", "data\_points" =\> { "TraficoEntrante" =\> 2474864615, "hostname" =\> "host1", "TraficoSaliente" =\> 247486… --- ## [Cloudtrail integration in logstash 2.4](https://discuss.elastic.co/t/cloudtrail-integration-in-logstash-2-4/95665)
**Author:** [@vijjun123](https://discuss.elastic.co/u/vijjun123)\ **Replies:** 1\ **Last updated:** [August 6, 2017, 6:54pm UTC](https://discuss.elastic.co/t/cloudtrail-integration-in-logstash-2-4/95665 "2017-08-06T18:54:05Z")
Hi All, Facing issue while creating logstash for cloudtrail integration and data is not flowing to elasticsearch. Please find the logstash 2.4 config details below. input{ s3 { bucket =\> "sademocloudtrail" sincedb… --- ## [Logstash dissect convert\_datatype](https://discuss.elastic.co/t/logstash-dissect-convert-datatype/96017)
**Author:** [@sockaddr\_in](https://discuss.elastic.co/u/sockaddr_in)\ **Replies:** 0\ **Last updated:** [August 6, 2017, 1:16pm UTC](https://discuss.elastic.co/t/logstash-dissect-convert-datatype/96017 "2017-08-06T13:16:00Z")
大家好： 我使用logstash的dissect插件进行nginx日志的字段切割，然后用convert\_datatype进行日志字段数据类型转换，但是有个问题，request\_time和upstream\_response\_time字段有时候并不一定是浮点类型，有时候却是'-'字符来代替，因此用convert\_datatype来将这两个字段转换成float类型时会抛异常，然后将这两个字段的值设置为原先的字符串，即'-'。我想要的结果是当… --- ## [Logstash filter if \[type\] condition ignored](https://discuss.elastic.co/t/logstash-filter-if-type-condition-ignored/96016)
**Author:** [@Architha](https://discuss.elastic.co/u/Architha)\ **Replies:** 0\ **Last updated:** [August 6, 2017, 12:30pm UTC](https://discuss.elastic.co/t/logstash-filter-if-type-condition-ignored/96016 "2017-08-06T12:30:27Z")
Hi All, I need to run multiple logstash config files on a single instance, for which i need to use, in each config file types. Each config file must have a different type or else the config files will step on each other… --- ## [In dissect plugin, use convert\_datatype to convert data type](https://discuss.elastic.co/t/in-dissect-plugin-use-convert-datatype-to-convert-data-type/96015)
**Author:** [@sockaddr\_in](https://discuss.elastic.co/u/sockaddr_in)\ **Replies:** 0\ **Last updated:** [August 6, 2017, 10:59am UTC](https://discuss.elastic.co/t/in-dissect-plugin-use-convert-datatype-to-convert-data-type/96015 "2017-08-06T10:59:07Z")
Hello, everyone! I logstash, i use dissect plugin to split nginx log, then use convert\_datatype to convert filed data type. But in my nginx log, the request\_time and upstream\_response\_time fields will be '-' sometimes, … --- ## [Dynamic add\_field in metrics](https://discuss.elastic.co/t/dynamic-add-field-in-metrics/94926)
**Author:** [@costef](https://discuss.elastic.co/u/costef)\ **Replies:** 1\ **Last updated:** [August 5, 2017, 7:58pm UTC](https://discuss.elastic.co/t/dynamic-add-field-in-metrics/94926 "2017-08-05T19:58:41Z")
I’ve a logstash conf file like the following: input { generator { type =\> “generated” } } filter { metrics { type =\> "generated" meter =\> "foo" add\_tag =\> "metric" add\_field =\> \[ “foo”, “Hello world, from %{ho… --- ## [Define the type of a date -logstash](https://discuss.elastic.co/t/define-the-type-of-a-date-logstash/94931)
**Author:** [@marwa](https://discuss.elastic.co/u/marwa)\ **Replies:** 1\ **Last updated:** [August 5, 2017, 7:54pm UTC](https://discuss.elastic.co/t/define-the-type-of-a-date-logstash/94931 "2017-08-05T19:54:44Z")
hi i trying to load csv file from logstash to elasticsearch.this is a sample of the structure of my data transaction\_date,customer\_name,product\_name,product\_id,price\_unit,quantity,fidelity\_card\_id,discount 2-24-2017,Ro… --- ## [Index documents daily from updated file](https://discuss.elastic.co/t/index-documents-daily-from-updated-file/94898)
**Author:** [@daaakeee](https://discuss.elastic.co/u/daaakeee)\ **Replies:** 1\ **Last updated:** [August 5, 2017, 7:53pm UTC](https://discuss.elastic.co/t/index-documents-daily-from-updated-file/94898 "2017-08-05T19:53:00Z")
Hi there! I wanted to learn, how can I index documents daily from newer file For example, I have file search\_2017.07.27.log input { file { path =\> \['/etc/logstash/elk\_searchlogs/search\_\*'\] start\_position =\> "beginn… --- ## [Date parsing can't update timestamp](https://discuss.elastic.co/t/date-parsing-cant-update-timestamp/94893)
**Author:** [@Yaohua](https://discuss.elastic.co/u/Yaohua)\ **Replies:** 1\ **Last updated:** [August 5, 2017, 6:27pm UTC](https://discuss.elastic.co/t/date-parsing-cant-update-timestamp/94893 "2017-08-05T18:27:27Z")
the date parsing is successful,but the @timestamp not updated This is my logstash input { stdin { codec =\> json} beats { port =\> 5043 } } filter { grok { patterns\_dir =\> \["/etc/logstash/pat… --- ## [Convert curl command to logstash-output-http](https://discuss.elastic.co/t/convert-curl-command-to-logstash-output-http/94871)
**Author:** [@lynchnf](https://discuss.elastic.co/u/lynchnf)\ **Replies:** 1\ **Last updated:** [August 5, 2017, 6:15pm UTC](https://discuss.elastic.co/t/convert-curl-command-to-logstash-output-http/94871 "2017-08-05T18:15:28Z")
I’m trying to replicate a curl command with Logstash 5.2.2 on a computer running Windows. The curl command is: curl -k -u "lynchnf:swordfish" https://mycompany.com:9998/some/service -d '{"event": {"some":"random","json… --- ## [XML parsing logstash, adding part of XML tags to an array per event](https://discuss.elastic.co/t/xml-parsing-logstash-adding-part-of-xml-tags-to-an-array-per-event/94827)
**Author:** [@g\_dev](https://discuss.elastic.co/u/g_dev)\ **Replies:** 1\ **Last updated:** [August 5, 2017, 6:03pm UTC](https://discuss.elastic.co/t/xml-parsing-logstash-adding-part-of-xml-tags-to-an-array-per-event/94827 "2017-08-05T18:03:45Z")
I'm trying to parsing xml file using multiline, and inside the main tag(from mutiline pattern) I want another tag to add to an array Issue --\> I'm not able to get \ into an array and later I can … --- ## [Logstash memory optimisations](https://discuss.elastic.co/t/logstash-memory-optimisations/94892)
**Author:** [@jaskaran\_singh](https://discuss.elastic.co/u/jaskaran_singh)\ **Replies:** 1\ **Last updated:** [August 5, 2017, 5:53pm UTC](https://discuss.elastic.co/t/logstash-memory-optimisations/94892 "2017-08-05T17:53:36Z")
Hi, Is there any benchmark for logstash memory requirements? Also, does removing unused plugins from the system help in the memory footprint of the logstash application? Thanks, Jaskaran [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=930) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=932)