# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=702 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 703 --- ## [Logstash 5.2 and specifying primitive types](https://discuss.elastic.co/t/logstash-5-2-and-specifying-primitive-types/144384)
**Author:** [@brandondash](https://discuss.elastic.co/u/brandondash)\ **Replies:** 2\ **Last updated:** [August 14, 2018, 6:50pm UTC](https://discuss.elastic.co/t/logstash-5-2-and-specifying-primitive-types/144384 "2018-08-14T18:50:46Z")
I have a pipeline that uses a grok pattern to force a particular field to be an integer. MY\_PATTERN ^%{DATE\_STAMP:for\_date} %{INT:total\_count:int}$ Logstash correctly parses my log and shoves the appropriate data into … --- ## [Filebeat with LogstashXML parser](https://discuss.elastic.co/t/filebeat-with-logstashxml-parser/143872)
**Author:** [@leggrand](https://discuss.elastic.co/u/leggrand)\ **Replies:** 7\ **Last updated:** [August 14, 2018, 6:18pm UTC](https://discuss.elastic.co/t/filebeat-with-logstashxml-parser/143872 "2018-08-14T18:18:10Z")
I create a template, with field maps, but the fields is empty. My logstash config is: input { beats { port =\> 5044 client\_inactivity\_timeout =\> 900 } } filter { xml { source =\> "message" … --- ## [Date filter ISO 8601](https://discuss.elastic.co/t/date-filter-iso-8601/144365)
**Author:** [@Ken\_Leoni](https://discuss.elastic.co/u/Ken_Leoni)\ **Replies:** 2\ **Last updated:** [August 14, 2018, 5:47pm UTC](https://discuss.elastic.co/t/date-filter-iso-8601/144365 "2018-08-14T17:47:38Z")
I'm doing something which I thought should be simple.. but can't get it to work for the life of me.. I'm doing a jdbc query.. and my data has a sampletime in the record that I want to use the the timestamp.. this is wha… --- ## [ELK for static logs. Is it viable option?](https://discuss.elastic.co/t/elk-for-static-logs-is-it-viable-option/144225)
**Author:** [@randomfx](https://discuss.elastic.co/u/randomfx)\ **Replies:** 1\ **Last updated:** [August 14, 2018, 5:13pm UTC](https://discuss.elastic.co/t/elk-for-static-logs-is-it-viable-option/144225 "2018-08-14T17:13:21Z")
Hi guys, New to ELK stack. Is it possible to use ELK for static log analysis? I mean if have a large archive with logs which are all connected/has hierarchical structure (root-branch), can I configure ELK to group it … --- ## [Issues in parsing baracuda firewall](https://discuss.elastic.co/t/issues-in-parsing-baracuda-firewall/144244)
**Author:** [@Samvid\_Kulkarni](https://discuss.elastic.co/u/Samvid_Kulkarni)\ **Replies:** 2\ **Last updated:** [August 14, 2018, 4:57pm UTC](https://discuss.elastic.co/t/issues-in-parsing-baracuda-firewall/144244 "2018-08-14T16:57:19Z")
I am trying to parse Baracuda Web Filter and I am having compile error. Here is the log 2018-08-07 11:00:01||1||5||bwf||192.168.6.141 - - \[07/Aug/2018:11:00:00 -0700\] "CONNECT store-images.s-microsoft.com:443 HTTP/1.0"… --- ## [Sql\_last\_value](https://discuss.elastic.co/t/sql-last-value/144345)
**Author:** [@Olha1](https://discuss.elastic.co/u/Olha1)\ **Replies:** 2\ **Last updated:** [August 14, 2018, 2:57pm UTC](https://discuss.elastic.co/t/sql-last-value/144345 "2018-08-14T14:57:32Z")
Hi! I have query in my logstash.config file. Where field "where" is: where column."Id" \> :sql\_last\_value But, in result, after run this I have column."Id" \> '2018-08-13 17:02:41.489705+0300' Why logstash get data las… --- ## [Does logstash drop Events](https://discuss.elastic.co/t/does-logstash-drop-events/144236)
**Author:** [@saad](https://discuss.elastic.co/u/saad)\ **Replies:** 5\ **Last updated:** [August 14, 2018, 1:54pm UTC](https://discuss.elastic.co/t/does-logstash-drop-events/144236 "2018-08-14T13:54:01Z")
How I will make sure that all events that are coming to logstash is going out and there is no dropping events in logstash queue? I mean I am not losing events in logstash and all events are successfully emitted to elast… --- ## [Jvm.options - elasticsearch and logstash?](https://discuss.elastic.co/t/jvm-options-elasticsearch-and-logstash/141817)
**Author:** [@S\_Chase](https://discuss.elastic.co/u/S_Chase)\ **Replies:** 3\ **Last updated:** [August 14, 2018, 1:37pm UTC](https://discuss.elastic.co/t/jvm-options-elasticsearch-and-logstash/141817 "2018-08-14T13:37:54Z")
I was getting some JVM Heap errors and I was looking up on how to set all of that properly. I saw in ES documentation to set it at 50% of available memory (I have 8G total) up to 32G. But which one do I set - LS or ES … --- ## [Calculate difference of time betwwen two logs formats!](https://discuss.elastic.co/t/calculate-difference-of-time-betwwen-two-logs-formats/144295)
**Author:** [@med\_amine](https://discuss.elastic.co/u/med_amine)\ **Replies:** 1\ **Last updated:** [August 14, 2018, 12:05pm UTC](https://discuss.elastic.co/t/calculate-difference-of-time-betwwen-two-logs-formats/144295 "2018-08-14T12:05:03Z")
hello guys , i have used ruby filter to calculate difference of time betwwen two logs formats which are ; --- ## [Logstash output to file, from JSON field not exist / field is empty, about output format](https://discuss.elastic.co/t/logstash-output-to-file-from-json-field-not-exist-field-is-empty-about-output-format/144060)
**Author:** [@richard20427176](https://discuss.elastic.co/u/richard20427176)\ **Replies:** 1\ **Last updated:** [August 14, 2018, 11:18am UTC](https://discuss.elastic.co/t/logstash-output-to-file-from-json-field-not-exist-field-is-empty-about-output-format/144060 "2018-08-14T11:18:08Z")
I'm new in using logstash (6.3.2) to parse log data, I want to parse datas in json fmt from logs file to txt files. The steps of input and filter are ok, but there's some problem with output, below is the output section … --- ## [How to search concatenated string in the message](https://discuss.elastic.co/t/how-to-search-concatenated-string-in-the-message/144280)
**Author:** [@Satya](https://discuss.elastic.co/u/Satya)\ **Replies:** 1\ **Last updated:** [August 14, 2018, 11:13am UTC](https://discuss.elastic.co/t/how-to-search-concatenated-string-in-the-message/144280 "2018-08-14T11:13:03Z")
HI, In my logstash layer I am receiving the cf loggregator message in json format. Example: { "name" : "abc", "address" : "bangalore} If I want to search in my logstash grok filter, if "name":"abc" is present in the… --- ## [Is logstash vulnerable to zip-slip?](https://discuss.elastic.co/t/is-logstash-vulnerable-to-zip-slip/144333)
**Author:** [@electrofelix](https://discuss.elastic.co/u/electrofelix)\ **Replies:** 0\ **Last updated:** [August 14, 2018, 11:04am UTC](https://discuss.elastic.co/t/is-logstash-vulnerable-to-zip-slip/144333 "2018-08-14T11:04:59Z")
Hi, Similar to a question I asked in the Kibana forum (Is the plugin install exposed to zip-slip?) due to some requests by the local security team, looking into whether some of the software we use is exposed to zip-slip… --- ## [Invalid Configuration Error](https://discuss.elastic.co/t/invalid-configuration-error/144078)
**Author:** [@Nithani25](https://discuss.elastic.co/u/Nithani25)\ **Replies:** 13\ **Last updated:** [August 14, 2018, 9:16am UTC](https://discuss.elastic.co/t/invalid-configuration-error/144078 "2018-08-14T09:16:03Z")
Hi Team, I am working on parsing apache access logs along with error logs to elastic via logstash index and facing invlaid configuration error. I am copying my filebeat input and logstash input for reference. Filebeat … --- ## [Error starting Logstash on RHEL7.5](https://discuss.elastic.co/t/error-starting-logstash-on-rhel7-5/144200)
**Author:** [@ariemenschneider](https://discuss.elastic.co/u/ariemenschneider)\ **Replies:** 2\ **Last updated:** [August 14, 2018, 7:59am UTC](https://discuss.elastic.co/t/error-starting-logstash-on-rhel7-5/144200 "2018-08-14T07:59:00Z")
Hello, I'm getting the following error when I try to start a newly configured logstash instance: \[2018-08-13T17:17:04,939\]\[ERROR\]\[org.logstash.Logstash \] java.lang.IllegalStateException: Logstash stopped processing … --- ## [ALB-Problem mapping types when importaing CSV via Logstash](https://discuss.elastic.co/t/alb-problem-mapping-types-when-importaing-csv-via-logstash/144117)
**Author:** [@alberto.latorre](https://discuss.elastic.co/u/alberto.latorre)\ **Replies:** 2\ **Last updated:** [August 14, 2018, 6:53am UTC](https://discuss.elastic.co/t/alb-problem-mapping-types-when-importaing-csv-via-logstash/144117 "2018-08-14T06:53:24Z")
Hello, all. I have a problem when converting or forcing a the types of csv files. I want to import into Elasticsearch using Logstash the following file: 12/08/2018-17:56:58,4802,34.25.128.24,Zapatillas de deporte 12/… --- ## [Aggregate filter with timeout\_timestamp\_field](https://discuss.elastic.co/t/aggregate-filter-with-timeout-timestamp-field/144260)
**Author:** [@lakshmi-github](https://discuss.elastic.co/u/lakshmi-github)\ **Replies:** 2\ **Last updated:** [August 14, 2018, 5:18am UTC](https://discuss.elastic.co/t/aggregate-filter-with-timeout-timestamp-field/144260 "2018-08-14T05:18:52Z")
Hello, I am trying to get a count of certain log events aggregated by minute where the timestamp should be based on a date-time string in the log. Based on the documentation for aggregation, here is what I have so far. … --- ## [Scandinavian letters from PostgreSQL](https://discuss.elastic.co/t/scandinavian-letters-from-postgresql/144274)
**Author:** [@MikkoLuhtasaari](https://discuss.elastic.co/u/MikkoLuhtasaari)\ **Replies:** 0\ **Last updated:** [August 14, 2018, 4:00am UTC](https://discuss.elastic.co/t/scandinavian-letters-from-postgresql/144274 "2018-08-14T04:00:12Z")
I'm trying to index data from PostgreSQL to Elasticsearch using Logstash. The database is encoded in UTF-8. Here's an example of data: nameöäå First name with å Second name with ä Third name with ö But when I'm tr… --- ## [Snmptrap input plugin fails as Error parsing xml with XmlSimple](https://discuss.elastic.co/t/snmptrap-input-plugin-fails-as-error-parsing-xml-with-xmlsimple/144058)
**Author:** [@paano](https://discuss.elastic.co/u/paano)\ **Replies:** 1\ **Last updated:** [August 14, 2018, 2:42am UTC](https://discuss.elastic.co/t/snmptrap-input-plugin-fails-as-error-parsing-xml-with-xmlsimple/144058 "2018-08-14T02:42:58Z")
Using Elastic/logstash version 6.2.3 After configuring SNMPTRAP, while the trap is generated, seeing error as 2018-08-12T20:04:53,651\]\[WARN \]\[logstash.filters.xml \] Error parsing xml with XmlSimple {:source=\>"messa… --- ## [ORA-01861 : literal does not match format string](https://discuss.elastic.co/t/ora-01861-literal-does-not-match-format-string/144263)
**Author:** [@seungtae](https://discuss.elastic.co/u/seungtae)\ **Replies:** 8\ **Last updated:** [August 14, 2018, 1:35am UTC](https://discuss.elastic.co/t/ora-01861-literal-does-not-match-format-string/144263 "2018-08-14T01:35:59Z")
Hello everyone. I want to convert string to date. My config, I use jdbc output plugin. jdbc { ... statement =\> \["INSERT INTO TB\_REN016\_020 (GATH\_DTM) VALUES(… --- ## [Consumer thredas](https://discuss.elastic.co/t/consumer-thredas/144262)
**Author:** [@wakame](https://discuss.elastic.co/u/wakame)\ **Replies:** 0\ **Last updated:** [August 14, 2018, 12:25am UTC](https://discuss.elastic.co/t/consumer-thredas/144262 "2018-08-14T00:25:08Z")
I am wondering how many consumer threads I need for the logstash. I have 1 topic in Kafka with 15 partitions. I have 6 logstash vm with 3 consumer threads. I’m doing around 100 million transactions a day. How many will b… --- ## [Creating Logstash metrics beyond counts, like sum or avg on field values of processed documents](https://discuss.elastic.co/t/creating-logstash-metrics-beyond-counts-like-sum-or-avg-on-field-values-of-processed-documents/144219)
**Author:** [@mbreest](https://discuss.elastic.co/u/mbreest)\ **Replies:** 2\ **Last updated:** [August 13, 2018, 8:00pm UTC](https://discuss.elastic.co/t/creating-logstash-metrics-beyond-counts-like-sum-or-avg-on-field-values-of-processed-documents/144219 "2018-08-13T20:00:52Z")
Hi folks, I have a use case, where I process log files from my CDN provider and want to count requests as well as the sum of bytes (bytes taken from log line) for the different products I have (set of domains or referre… --- ## [XML Filter :How to create array of objects using xml filter (xpath)](https://discuss.elastic.co/t/xml-filter-how-to-create-array-of-objects-using-xml-filter-xpath/144143)
**Author:** [@Muhammed\_Aboobacker](https://discuss.elastic.co/u/Muhammed_Aboobacker)\ **Replies:** 3\ **Last updated:** [August 13, 2018, 6:20pm UTC](https://discuss.elastic.co/t/xml-filter-how-to-create-array-of-objects-using-xml-filter-xpath/144143 "2018-08-13T18:20:39Z")
I am new to logstash. I am trying push XML (generated as the output of http polling) in logstash. I want to have the XML elements attributes convert to json objects.I am using xpath for parsing xml data.But i am getting… --- ## [Upload the cSV multiple times](https://discuss.elastic.co/t/upload-the-csv-multiple-times/144133)
**Author:** [@ramnar125](https://discuss.elastic.co/u/ramnar125)\ **Replies:** 1\ **Last updated:** [August 13, 2018, 5:56pm UTC](https://discuss.elastic.co/t/upload-the-csv-multiple-times/144133 "2018-08-13T17:56:08Z")
Hi , i am planning to load a CSV file with a set of data in logstash. then after 1 hour i want to upload the same CSV file again. so let me know how to delete the old set of data in logstash and upload the new CSV so … --- ## [Creating separate documents from xml file : \_split\_type\_failure](https://discuss.elastic.co/t/creating-separate-documents-from-xml-file-split-type-failure/144205)
**Author:** [@safaa](https://discuss.elastic.co/u/safaa)\ **Replies:** 3\ **Last updated:** [August 13, 2018, 5:47pm UTC](https://discuss.elastic.co/t/creating-separate-documents-from-xml-file-split-type-failure/144205 "2018-08-13T17:47:37Z")
Hi, i parse xml file that contains 500 tag named" ReportItem ". I am using split filter to add disctint documents for each ReportItem . Logstash only parse the 3 documents and return file \_split\_type\_failure. Here is my… --- ## [Logstash Windows Service - error java.lang.IllegalStateException](https://discuss.elastic.co/t/logstash-windows-service-error-java-lang-illegalstateexception/144187)
**Author:** [@Scott\_Newby](https://discuss.elastic.co/u/Scott_Newby)\ **Replies:** 4\ **Last updated:** [August 13, 2018, 5:45pm UTC](https://discuss.elastic.co/t/logstash-windows-service-error-java-lang-illegalstateexception/144187 "2018-08-13T17:45:15Z")
Logstash continues to restart repeating - \[ERROR\]\[org.logstash.Logstash \] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit I've tried: removing the config.reload.au… --- ## [Logstash 6.2.4+ significantly slower than 6.2.3](https://discuss.elastic.co/t/logstash-6-2-4-significantly-slower-than-6-2-3/144214)
**Author:** [@jong99](https://discuss.elastic.co/u/jong99)\ **Replies:** 1\ **Last updated:** [August 13, 2018, 5:23pm UTC](https://discuss.elastic.co/t/logstash-6-2-4-significantly-slower-than-6-2-3/144214 "2018-08-13T17:23:48Z")
I've had some issues with Logstash 6 seeming very slow so I decided to benchmark different versions using the benchmark tool at https://github.com/elastic/logstash/tree/master/tools/benchmark-cli If I compare the Throug… --- ## [Move subarrays to document root](https://discuss.elastic.co/t/move-subarrays-to-document-root/143876)
**Author:** [@programagor](https://discuss.elastic.co/u/programagor)\ **Replies:** 7\ **Last updated:** [August 13, 2018, 3:06pm UTC](https://discuss.elastic.co/t/move-subarrays-to-document-root/143876 "2018-08-13T15:06:03Z")
Greetings My incoming messages contain subarrays in the following format: com.openexchange.ajax.action com.openexchange.ajax.module com.openexchange.database.schema com.openexchange.grizzly.method com.openexchange.griz… --- ## [Cloudtrail logs using logstash](https://discuss.elastic.co/t/cloudtrail-logs-using-logstash/143955)
**Author:** [@jellsworth](https://discuss.elastic.co/u/jellsworth)\ **Replies:** 5\ **Last updated:** [August 13, 2018, 2:04pm UTC](https://discuss.elastic.co/t/cloudtrail-logs-using-logstash/143955 "2018-08-13T14:04:34Z")
I have a mostly-working config for ingesting my cloudtrail logs using logstash. ElasticSearch doesn't seem to be parsing the 'ResponseElements' field, though, as well as a few other fields that can vary from being empty … --- ## [Why geoip filter d'ont work for some IP address?](https://discuss.elastic.co/t/why-geoip-filter-dont-work-for-some-ip-address/144174)
**Author:** [@med\_amine](https://discuss.elastic.co/u/med_amine)\ **Replies:** 2\ **Last updated:** [August 13, 2018, 1:22pm UTC](https://discuss.elastic.co/t/why-geoip-filter-dont-work-for-some-ip-address/144174 "2018-08-13T13:22:58Z")
hello guys , i have used geoip filter in my configuration , i notice that this filter d'ont work for some IP address ; for example ; 154.107.183.56 ; geoip " work " 172.22.1.53 ; not working if this normal or t… --- ## [Log only file metadata not content](https://discuss.elastic.co/t/log-only-file-metadata-not-content/144173)
**Author:** [@sujeetKumar](https://discuss.elastic.co/u/sujeetKumar)\ **Replies:** 1\ **Last updated:** [August 13, 2018, 1:19pm UTC](https://discuss.elastic.co/t/log-only-file-metadata-not-content/144173 "2018-08-13T13:19:29Z")
Our requirement is just to get file metadata (file names, createdDate, lastmodified etc) from a path. We have a directory where all the stdout and stderr files are being stored for different jobs. We just want to log th… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=701) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=703)