# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=663 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 664 --- ## [Splitting each element(s) in nested array field into 2 respectively](https://discuss.elastic.co/t/splitting-each-element-s-in-nested-array-field-into-2-respectively/153515)
**Author:** [@dx123](https://discuss.elastic.co/u/dx123)\ **Replies:** 0\ **Last updated:** [October 23, 2018, 4:19am UTC](https://discuss.elastic.co/t/splitting-each-element-s-in-nested-array-field-into-2-respectively/153515 "2018-10-23T04:19:22Z")
I've found a few threads online discussing this but what my situation requires is more complex. Basically I have a array data field called target, where in most of the cases in my feeds it just consists of 1 element but … --- ## [Parse with Grok an HTTP communication](https://discuss.elastic.co/t/parse-with-grok-an-http-communication/153465)
**Author:** [@camay123](https://discuss.elastic.co/u/camay123)\ **Replies:** 1\ **Last updated:** [October 23, 2018, 1:25am UTC](https://discuss.elastic.co/t/parse-with-grok-an-http-communication/153465 "2018-10-23T01:25:08Z")
I am trying to parse an HTTP header communication with grok/logstash, here is a sample comm: GET /msdownload/update/v3/static/trustedr/en/authrootstl.cab?07f093d6b2ea8d03 HTTP/1.1 Connection: Keep-Alive Accept: \*/\* User… --- ## [Splunk API monitor - Logstash pipeline question?](https://discuss.elastic.co/t/splunk-api-monitor-logstash-pipeline-question/153502)
**Author:** [@bloke](https://discuss.elastic.co/u/bloke)\ **Replies:** 0\ **Last updated:** [October 23, 2018, 12:43am UTC](https://discuss.elastic.co/t/splunk-api-monitor-logstash-pipeline-question/153502 "2018-10-23T00:43:22Z")
Hi Folks, I built a logstash pipeline to monitor splunk via the splunk API by doing the following. Build multi-file logstash pipeline like such 100-input-exec.conf input { exec { id =\> "cur… --- ## [Logstash insufficient memory for Java Runtime Environment](https://discuss.elastic.co/t/logstash-insufficient-memory-for-java-runtime-environment/153460)
**Author:** [@fionamacd](https://discuss.elastic.co/u/fionamacd)\ **Replies:** 5\ **Last updated:** [October 22, 2018, 6:44pm UTC](https://discuss.elastic.co/t/logstash-insufficient-memory-for-java-runtime-environment/153460 "2018-10-22T18:44:33Z")
After adding the filebeat agents on a number of servers I am seeing memory errors in Logstash - I was just trying to get the logstash version to make sure I had the current one and get this error: ubuntu@ip-10-152-3-25:… --- ## [Filtering data from a "message" field in a .conf file for logstash](https://discuss.elastic.co/t/filtering-data-from-a-message-field-in-a-conf-file-for-logstash/153458)
**Author:** [@mahon2dm](https://discuss.elastic.co/u/mahon2dm)\ **Replies:** 0\ **Last updated:** [October 22, 2018, 6:20pm UTC](https://discuss.elastic.co/t/filtering-data-from-a-message-field-in-a-conf-file-for-logstash/153458 "2018-10-22T18:20:57Z")
I am having a problem writing a correct .conf file for Logstash to ingest into ElasticSearch. Currently I have a "Conversational\_history.log" file from a Chatbot application that I have linked up to Filebeat, and from Fi… --- ## [Django logstash filter](https://discuss.elastic.co/t/django-logstash-filter/149246)
**Author:** [@esseti](https://discuss.elastic.co/u/esseti)\ **Replies:** 7\ **Last updated:** [October 22, 2018, 6:13pm UTC](https://discuss.elastic.co/t/django-logstash-filter/149246 "2018-10-22T18:13:33Z")
HI all, is there an "official" or standard-de-facto filter for logstash that can be used in Django? For what I know it should extend SocketHandler (TCP) or DatagramHandler (UDP), but I would love to build something fro… --- ## [Backup and create new data](https://discuss.elastic.co/t/backup-and-create-new-data/152882)
**Author:** [@shani\_angarkadu](https://discuss.elastic.co/u/shani_angarkadu)\ **Replies:** 3\ **Last updated:** [October 22, 2018, 6:09pm UTC](https://discuss.elastic.co/t/backup-and-create-new-data/152882 "2018-10-22T18:09:38Z")
hi community, we are using the log-stash to import the data from oracle db everyday to elastic. what would be the best workflow and steps to automate the job? Current steps,each runs individually, Is there anyway to m… --- ## [Gsub filter not working as expected for dot to underscore conversion](https://discuss.elastic.co/t/gsub-filter-not-working-as-expected-for-dot-to-underscore-conversion/152180)
**Author:** [@Gauti](https://discuss.elastic.co/u/Gauti)\ **Replies:** 3\ **Last updated:** [October 22, 2018, 4:55pm UTC](https://discuss.elastic.co/t/gsub-filter-not-working-as-expected-for-dot-to-underscore-conversion/152180 "2018-10-22T16:55:48Z")
Hi All, I have been trying to replace dots to underscores in the fieldname using gsub filter and didnt succeeded, here is my filter mutate { gsub =\> \[ "caller\_id.location.latitude", ".", "\_" \] } t… --- ## [Timestamp problem](https://discuss.elastic.co/t/timestamp-problem/153447)
**Author:** [@wheelq](https://discuss.elastic.co/u/wheelq)\ **Replies:** 0\ **Last updated:** [October 22, 2018, 4:37pm UTC](https://discuss.elastic.co/t/timestamp-problem/153447 "2018-10-22T16:37:45Z")
I got date saved as a string:"1538644629000". I need it to be saved as a timestamp so it can be used in Kibana. I have tried everything (I think): date{ match =\> \["temp\_date", "yyyy-MM-dd'T'HH:mm:ss'.'SSS'Z'"\] target … --- ## [Logstash unable to parse xml file](https://discuss.elastic.co/t/logstash-unable-to-parse-xml-file/153372)
**Author:** [@safaa](https://discuss.elastic.co/u/safaa)\ **Replies:** 2\ **Last updated:** [October 22, 2018, 3:45pm UTC](https://discuss.elastic.co/t/logstash-unable-to-parse-xml-file/153372 "2018-10-22T15:45:55Z")
I have a simple xml file , and i am trying to parse it with losgstash, but the logstash could not undersand split field: my xml file is: \ \ \Value1\ \Value2\ … --- ## [Logstash Conditionals Efficiency/Best Practise?](https://discuss.elastic.co/t/logstash-conditionals-efficiency-best-practise/153435)
**Author:** [@londonx](https://discuss.elastic.co/u/londonx)\ **Replies:** 0\ **Last updated:** [October 22, 2018, 3:29pm UTC](https://discuss.elastic.co/t/logstash-conditionals-efficiency-best-practise/153435 "2018-10-22T15:29:28Z")
Hi I am quite interested in using Logstash conditionals to lookup the server hostname in an array to determine the rest of the pipeline, giving me a single pipeline/shipper file that covers all servers/environments. A t… --- ## [Translate plugin does not populate destination if it already exists](https://discuss.elastic.co/t/translate-plugin-does-not-populate-destination-if-it-already-exists/153433)
**Author:** [@parosio](https://discuss.elastic.co/u/parosio)\ **Replies:** 0\ **Last updated:** [October 22, 2018, 3:26pm UTC](https://discuss.elastic.co/t/translate-plugin-does-not-populate-destination-if-it-already-exists/153433 "2018-10-22T15:26:25Z")
Hello, I think I've found a rather weird behavior of translate plugin (logstash 5.4). I'm looking for a match in a first dictionary; if no match is found, I need to use a second dictionary. The second translation only… --- ## [Mutate date inside json](https://discuss.elastic.co/t/mutate-date-inside-json/153427)
**Author:** [@wheelq](https://discuss.elastic.co/u/wheelq)\ **Replies:** 0\ **Last updated:** [October 22, 2018, 3:06pm UTC](https://discuss.elastic.co/t/mutate-date-inside-json/153427 "2018-10-22T15:06:27Z")
I got json array inside one of the fields I'm going to store using logstash, and inside this arraythere is another field which contains a timestamp. How do I access and mutate it? I have tried the following: date{ ma… --- ## [Logstash stopped working after upgrade 6.2.4 --\> 6.4.0](https://discuss.elastic.co/t/logstash-stopped-working-after-upgrade-6-2-4-6-4-0/146626)
**Author:** [@RonGros](https://discuss.elastic.co/u/RonGros)\ **Replies:** 15\ **Last updated:** [October 22, 2018, 2:58pm UTC](https://discuss.elastic.co/t/logstash-stopped-working-after-upgrade-6-2-4-6-4-0/146626 "2018-10-22T14:58:40Z")
Hi, Yesterday I tried migrating logstash from 6.2.4 to 6.4 because I've been waiting for the "Read" mode for ages!! then I found that logstash did nothing after I started it - the sincedb is being updated when logstash… --- ## [Error While Running Logstash - import data to Elastic](https://discuss.elastic.co/t/error-while-running-logstash-import-data-to-elastic/153424)
**Author:** [@safarhan](https://discuss.elastic.co/u/safarhan)\ **Replies:** 0\ **Last updated:** [October 22, 2018, 2:42pm UTC](https://discuss.elastic.co/t/error-while-running-logstash-import-data-to-elastic/153424 "2018-10-22T14:42:51Z")
Hello, When I run logstash -f sql.conf to import data from Microsoft SQL Server to Elastic, it gives the following error: Error: Could not find or load main class Driver I tried just logstash and it gave same error. I… --- ## [Aggregating json events](https://discuss.elastic.co/t/aggregating-json-events/153408)
**Author:** [@IC91](https://discuss.elastic.co/u/IC91)\ **Replies:** 1\ **Last updated:** [October 22, 2018, 2:05pm UTC](https://discuss.elastic.co/t/aggregating-json-events/153408 "2018-10-22T14:05:31Z")
Hey guys, So I am using HTTP poller input plugin to get Json formatted replies from two URLs. I want to aggregate the two events based on a common field. Is there any direct plugin to do it? Any help is appreciable. Tha… --- ## [Logstash6.4.1 docker container receives traps but doesn't process](https://discuss.elastic.co/t/logstash6-4-1-docker-container-receives-traps-but-doesnt-process/153384)
**Author:** [@elastic3](https://discuss.elastic.co/u/elastic3)\ **Replies:** 1\ **Last updated:** [October 22, 2018, 1:29pm UTC](https://discuss.elastic.co/t/logstash6-4-1-docker-container-receives-traps-but-doesnt-process/153384 "2018-10-22T13:29:47Z")
Hello, I am new to this forum, so please bear with me. I am using Logstash 6.4.1 image with a bit more changes using Dockerfile to customise it. I am using Logstash to receive SNMP trap from devices. Using tcpdump I … --- ## [Failed parsing date in format "dd/MM/yyyy" for specific date](https://discuss.elastic.co/t/failed-parsing-date-in-format-dd-mm-yyyy-for-specific-date/152312)
**Author:** [@rlapolli](https://discuss.elastic.co/u/rlapolli)\ **Replies:** 12\ **Last updated:** [October 22, 2018, 1:04pm UTC](https://discuss.elastic.co/t/failed-parsing-date-in-format-dd-mm-yyyy-for-specific-date/152312 "2018-10-22T13:04:17Z")
Hello everyone. I'm having trouble analyzing date for the "dd / mm / yyyy" format. I'm using the "Date Filter". Everything works perfectly for any date, except for "10/16/2016". Yes, it sounds quite weird, but I have t… --- ## [Conditional if on output using id](https://discuss.elastic.co/t/conditional-if-on-output-using-id/152128)
**Author:** [@agx](https://discuss.elastic.co/u/agx)\ **Replies:** 1\ **Last updated:** [October 22, 2018, 1:03pm UTC](https://discuss.elastic.co/t/conditional-if-on-output-using-id/152128 "2018-10-22T13:03:04Z")
If I set id =\> "my\_plugin\_id" on an input can i use: if \[id\] == "my\_plugin\_id" on output to ensure the correct output is used? --- ## [Logstash input Odata /API stream](https://discuss.elastic.co/t/logstash-input-odata-api-stream/152120)
**Author:** [@jawahar](https://discuss.elastic.co/u/jawahar)\ **Replies:** 1\ **Last updated:** [October 22, 2018, 1:01pm UTC](https://discuss.elastic.co/t/logstash-input-odata-api-stream/152120 "2018-10-22T13:01:44Z")
I want to use logstach to bring in data from a public OData site or even via a public API. Like the NYC 311 service requests data stream do I use an HTTP input, how do I specify the API call to the csv data https://da… --- ## [Can't load CSV data into elasticsearch index](https://discuss.elastic.co/t/cant-load-csv-data-into-elasticsearch-index/153048)
**Author:** [@elk\_enthu](https://discuss.elastic.co/u/elk_enthu)\ **Replies:** 1\ **Last updated:** [October 22, 2018, 12:22pm UTC](https://discuss.elastic.co/t/cant-load-csv-data-into-elasticsearch-index/153048 "2018-10-22T12:22:40Z")
Hi there, Since last 3 days, I’m trying to explore ELK stack and importing CSV table into Elasticsearch through Logstash. Checked many forums and videos and tried the following. But, I get various errors. Here’s what I’… --- ## [Logstash stops processing after some time](https://discuss.elastic.co/t/logstash-stops-processing-after-some-time/153111)
**Author:** [@meetdave2611997](https://discuss.elastic.co/u/meetdave2611997)\ **Replies:** 3\ **Last updated:** [October 22, 2018, 10:28am UTC](https://discuss.elastic.co/t/logstash-stops-processing-after-some-time/153111 "2018-10-22T10:28:05Z")
Hi, In our stack, we have 12 logstash nodes. They work very fine when we restart all of them. After some time some nodes stop accepting data and some keeps going and processing. This way all of the nodes are getting dow… --- ## [Can't find cgroup file, preventing pushing to an Elastic Cloud instance](https://discuss.elastic.co/t/cant-find-cgroup-file-preventing-pushing-to-an-elastic-cloud-instance/152744)
**Author:** [@Benjamin\_Roffey](https://discuss.elastic.co/u/Benjamin_Roffey)\ **Replies:** 1\ **Last updated:** [October 22, 2018, 10:19am UTC](https://discuss.elastic.co/t/cant-find-cgroup-file-preventing-pushing-to-an-elastic-cloud-instance/152744 "2018-10-22T10:19:46Z")
Hi All, I've been trying to setup a logstash export to the elastic cloud. I can parse the files into the correct format and send them to stdout, but when I try to send them to the cloud, I get the following error: //\[l… --- ## [Get piece of message from huge message](https://discuss.elastic.co/t/get-piece-of-message-from-huge-message/153367)
**Author:** [@npontes](https://discuss.elastic.co/u/npontes)\ **Replies:** 0\ **Last updated:** [October 22, 2018, 10:02am UTC](https://discuss.elastic.co/t/get-piece-of-message-from-huge-message/153367 "2018-10-22T10:02:59Z")
Hello everyone, I want to write a list of emails that are in the ES in a csv. However the emails are inside a huge message. The ES has a field named @request\_logline and when they exist they are in the following forma… --- ## [Schedule keyword not generating index in ES via JDBC-input plugin](https://discuss.elastic.co/t/schedule-keyword-not-generating-index-in-es-via-jdbc-input-plugin/153184)
**Author:** [@groverjatin17](https://discuss.elastic.co/u/groverjatin17)\ **Replies:** 1\ **Last updated:** [October 22, 2018, 9:09am UTC](https://discuss.elastic.co/t/schedule-keyword-not-generating-index-in-es-via-jdbc-input-plugin/153184 "2018-10-22T09:09:38Z")
Hello All, Please help me solve this i need to deliver this to my manager. I am trying to connect to the database using Logstash-jdbc-input plugin. To keep the data in Kibana up to date with the table feeding it I ha… --- ## [Getting error INVALID\_FETCH\_SESSION\_EPOCH in logstash still](https://discuss.elastic.co/t/getting-error-invalid-fetch-session-epoch-in-logstash-still/153352)
**Author:** [@nickytd](https://discuss.elastic.co/u/nickytd)\ **Replies:** 0\ **Last updated:** [October 22, 2018, 8:59am UTC](https://discuss.elastic.co/t/getting-error-invalid-fetch-session-epoch-in-logstash-still/153352 "2018-10-22T08:59:38Z")
Hi Occasionally log stash Kafka consumer produces \[INFO \]\[org.apache.kafka.clients.FetchSessionHandler\] \[Consumer clientId=elk-logstash-indexer-69bb94c58-4x7q8-4, groupId=containers\] Node 1 was unable to process the fe… --- ## [How to combine multiple XML elements?](https://discuss.elastic.co/t/how-to-combine-multiple-xml-elements/153351)
**Author:** [@humalog](https://discuss.elastic.co/u/humalog)\ **Replies:** 0\ **Last updated:** [October 22, 2018, 8:57am UTC](https://discuss.elastic.co/t/how-to-combine-multiple-xml-elements/153351 "2018-10-22T08:57:06Z")
I'm trying to split XML file in multiple events. Here's a sample input: \ \ \ \linux\ \Linux Kernel 2.6\ **Author:** [@Werring](https://discuss.elastic.co/u/Werring)\ **Replies:** 2\ **Last updated:** [October 22, 2018, 8:52am UTC](https://discuss.elastic.co/t/filebeats-input-json-lines-only-processing-single-entry/153348 "2018-10-22T08:52:09Z") I am just starting with logstash (and the other products of the elastic stack) but im having some issues with trying to set up my sample parser. Only the first line of my log is sent to the stdout when using the json\_li… --- ## [Logstash Event Limit](https://discuss.elastic.co/t/logstash-event-limit/152928)
**Author:** [@peter.alfafara](https://discuss.elastic.co/u/peter.alfafara)\ **Replies:** 9\ **Last updated:** [October 22, 2018, 5:26am UTC](https://discuss.elastic.co/t/logstash-event-limit/152928 "2018-10-22T05:26:10Z")
Hi Experts, I'd like to inquire about logstash limits. I have an event coming in to logstash with about 6700 records however it doesn't process any of these and ignore it. Is there any configuration I can adjust in ord… --- ## [How can I add this ruby code?](https://discuss.elastic.co/t/how-can-i-add-this-ruby-code/152580)
**Author:** [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Replies:** 4\ **Last updated:** [October 21, 2018, 11:33pm UTC](https://discuss.elastic.co/t/how-can-i-add-this-ruby-code/152580 "2018-10-21T23:33:10Z")
Hi, I have GPS coordinates in DMS (degree minute second) that I need to convert to DD (decimal degrees) so they can be used in Elastic as geo points. I already wrote I probably not very good filter to convert South and… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=662) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=664)