# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=546 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 547 --- ## [Exclude ordered list numbers while parsing](https://discuss.elastic.co/t/exclude-ordered-list-numbers-while-parsing/181593)
**Author:** [@Abhilash\_B](https://discuss.elastic.co/u/Abhilash_B)\ **Replies:** 5\ **Last updated:** [May 17, 2019, 12:48pm UTC](https://discuss.elastic.co/t/exclude-ordered-list-numbers-while-parsing/181593 "2019-05-17T12:48:25Z")
Hi all, I have 1. StartTime:3761976079 to be parsed, it may have leading and trailing spaces. I am using this grok expression to parse ^(?\\[^:\]+):\\s\*%{GREEDYDATA:value}. How do I exclude the number at the start? Cur… --- ## [Logstash-elasticsearch design question: process XML payloads from database](https://discuss.elastic.co/t/logstash-elasticsearch-design-question-process-xml-payloads-from-database/181522)
**Author:** [@Kurt\_V](https://discuss.elastic.co/u/Kurt_V)\ **Replies:** 1\ **Last updated:** [May 17, 2019, 12:45pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-design-question-process-xml-payloads-from-database/181522 "2019-05-17T12:45:38Z")
Hi, I've been given the task to rewrite an application that searches for audit data in an SQL Sever database. Currently the data is being queried using regular SQL queries, but I would like to go for a better approach, … --- ## [Logstash to support nanosecond timestamps?](https://discuss.elastic.co/t/logstash-to-support-nanosecond-timestamps/181584)
**Author:** [@akowalski](https://discuss.elastic.co/u/akowalski)\ **Replies:** 0\ **Last updated:** [May 17, 2019, 11:36am UTC](https://discuss.elastic.co/t/logstash-to-support-nanosecond-timestamps/181584 "2019-05-17T11:36:37Z")
Hi, With elastic 7.0 release notes I was really happy to see native support for nanosecond timestamps. How does this work with Logstash? I see the date filter plugin sill supports only UNIX and UNIX\_MS input format. I … --- ## [Event Hub failed during initialization](https://discuss.elastic.co/t/event-hub-failed-during-initialization/181437)
**Author:** [@eweitkam](https://discuss.elastic.co/u/eweitkam)\ **Replies:** 1\ **Last updated:** [May 17, 2019, 10:43am UTC](https://discuss.elastic.co/t/event-hub-failed-during-initialization/181437 "2019-05-17T10:43:30Z")
Hi, I get a error that the logstash.inputs.azureeventhubs could not be initialized. I am using version 7.0.1 with the default installed plugin: logstash-input-azure\_event\_hubs Complaining about undefined method \`getE… --- ## [Xml plugin getting text with characters](https://discuss.elastic.co/t/xml-plugin-getting-text-with-characters/180389)
**Author:** [@bigster](https://discuss.elastic.co/u/bigster)\ **Replies:** 10\ **Last updated:** [May 17, 2019, 10:02am UTC](https://discuss.elastic.co/t/xml-plugin-getting-text-with-characters/180389 "2019-05-17T10:02:34Z")
Hi, I'm getting the XML values from a QUEUE and when i get an element that have the '&' character it transforms into the & string. Here are the before and after: gz$LDQ@vnbkj7DNUVwwjrnGa0&|2,a) gz$LDQ@vnbkj7DNU… --- ## [How to read textfiles dynamically from folder and keep listining without jump out Heap Size Memory](https://discuss.elastic.co/t/how-to-read-textfiles-dynamically-from-folder-and-keep-listining-without-jump-out-heap-size-memory/181557)
**Author:** [@samyo](https://discuss.elastic.co/u/samyo)\ **Replies:** 0\ **Last updated:** [May 17, 2019, 9:38am UTC](https://discuss.elastic.co/t/how-to-read-textfiles-dynamically-from-folder-and-keep-listining-without-jump-out-heap-size-memory/181557 "2019-05-17T09:38:44Z")
Hello folks, i am using Elasticsearch and Logstash Version 6.3.1 , and Java version 8 , on Ubunto Virtualmachine , with 8GB RAM and 50GB for the Memory . I am trying to read and match 8000 textfiles from folder using m… --- ## [Logstash connections to ES](https://discuss.elastic.co/t/logstash-connections-to-es/181555)
**Author:** [@Tuckson](https://discuss.elastic.co/u/Tuckson)\ **Replies:** 0\ **Last updated:** [May 17, 2019, 9:34am UTC](https://discuss.elastic.co/t/logstash-connections-to-es/181555 "2019-05-17T09:34:14Z")
Hi, Running elkstack 6.6 here. I have these 3 logstash servers which all 3 connect to 1 coördinating node in my 6 node ES cluster (I understood this is the most performant way of doing this). All my application server… --- ## [Xml filtering subfields](https://discuss.elastic.co/t/xml-filtering-subfields/177436)
**Author:** [@nane96](https://discuss.elastic.co/u/nane96)\ **Replies:** 20\ **Last updated:** [May 15, 2019, 10:39pm UTC](https://discuss.elastic.co/t/xml-filtering-subfields/177436 "2019-05-15T22:39:58Z")
Hey! I'm a complete beginner to everything elasticsearch/logstash/kibana related, so please have mercy. I have a xml-File with many Eventlogs that is (simplified) structured like this: \ \ … --- ## [Filter cef](https://discuss.elastic.co/t/filter-cef/181215)
**Author:** [@ta7643](https://discuss.elastic.co/u/ta7643)\ **Replies:** 6\ **Last updated:** [May 17, 2019, 9:09am UTC](https://discuss.elastic.co/t/filter-cef/181215 "2019-05-17T09:09:09Z")
Hi, in the logstash reference is no cef filter plugin but there is a codec cef which i use for my input. input { stdin { codec =\> cef } } filter {} output { stdout { codec =\> rubydebug } } This is working fine but i w… --- ## [How to handle multiple syslog inputs with Logstash to different indices](https://discuss.elastic.co/t/how-to-handle-multiple-syslog-inputs-with-logstash-to-different-indices/181414)
**Author:** [@colix](https://discuss.elastic.co/u/colix)\ **Replies:** 4\ **Last updated:** [May 17, 2019, 8:23am UTC](https://discuss.elastic.co/t/how-to-handle-multiple-syslog-inputs-with-logstash-to-different-indices/181414 "2019-05-17T08:23:58Z")
Hi, in input I use 2 ports by 2 project, I want handle them to different indices. But now all input go to prd.rnis-\* index. Can you help? input { syslog { port =\> 5001 tags =\> \["p… --- ## [Migrating Elastic Search Data Using Logstash](https://discuss.elastic.co/t/migrating-elastic-search-data-using-logstash/181514)
**Author:** [@Thilak](https://discuss.elastic.co/u/Thilak)\ **Replies:** 0\ **Last updated:** [May 17, 2019, 6:02am UTC](https://discuss.elastic.co/t/migrating-elastic-search-data-using-logstash/181514 "2019-05-17T06:02:26Z")
input { ElasticSearch { hosts =\> \["\*\*\*\*\*\*\*\*your host\*\*\*\*\*\*\*\*\*\*"\] user =\> "\*\*\*\*\*\*\*" password =\> "\*\*\*\*\*\*\*\*\*" index =\> "logstash-2017.11.07" size =\> 1000 scroll =\> "1m" … --- ## [Logstash-output-email-plugin not working](https://discuss.elastic.co/t/logstash-output-email-plugin-not-working/181513)
**Author:** [@cppatel](https://discuss.elastic.co/u/cppatel)\ **Replies:** 0\ **Last updated:** [May 17, 2019, 5:25am UTC](https://discuss.elastic.co/t/logstash-output-email-plugin-not-working/181513 "2019-05-17T05:25:20Z")
Hi all, When I try to install the logstash-output-email-4.1.1.zip plugin it's give me following error. I have downloaded it directly from github. It also includes gem to run. Please help me with the installation. Insta… --- ## [One logstash file with multiple logs from different nodes](https://discuss.elastic.co/t/one-logstash-file-with-multiple-logs-from-different-nodes/181476)
**Author:** [@George\_Smiley](https://discuss.elastic.co/u/George_Smiley)\ **Replies:** 0\ **Last updated:** [May 16, 2019, 8:19pm UTC](https://discuss.elastic.co/t/one-logstash-file-with-multiple-logs-from-different-nodes/181476 "2019-05-16T20:19:03Z")
Hi, I am trying to build a centralized log server using elk stack. In my cluster, there is one master node which has logstash, elasticsearch, kibana on it. And I am trying to use filebeat on each of my server node to se… --- ## [Convert date to string string](https://discuss.elastic.co/t/convert-date-to-string-string/181128)
**Author:** [@Yoav\_Ben\_Moha](https://discuss.elastic.co/u/Yoav_Ben_Moha)\ **Replies:** 11\ **Last updated:** [May 16, 2019, 7:57pm UTC](https://discuss.elastic.co/t/convert-date-to-string-string/181128 "2019-05-16T19:57:08Z")
Hi, I am using ELK7 . I have a csv file with 3 columns I have successfully converted the first two columns from string to integer , but failed to convert the 3rd column from string to date. In Kibana the 3rd field i… --- ## [Pipeline shows events being received, but index is empty](https://discuss.elastic.co/t/pipeline-shows-events-being-received-but-index-is-empty/181469)
**Author:** [@rstasiunas](https://discuss.elastic.co/u/rstasiunas)\ **Replies:** 0\ **Last updated:** [May 16, 2019, 6:54pm UTC](https://discuss.elastic.co/t/pipeline-shows-events-being-received-but-index-is-empty/181469 "2019-05-16T18:54:58Z")
Hello, I have a Logstash pipeline that is set to poll an AWS SQS queue. The events show in the queue, and the pipeline data in Kibana shows events being received, the index gets created, however, the index remains empt… --- ## [Unable to retreive data from index using elasticsearch input plugin in logstash](https://discuss.elastic.co/t/unable-to-retreive-data-from-index-using-elasticsearch-input-plugin-in-logstash/181375)
**Author:** [@Vishnu\_mk](https://discuss.elastic.co/u/Vishnu_mk)\ **Replies:** 6\ **Last updated:** [May 16, 2019, 5:11pm UTC](https://discuss.elastic.co/t/unable-to-retreive-data-from-index-using-elasticsearch-input-plugin-in-logstash/181375 "2019-05-16T17:11:51Z")
input { elasticsearch { hosts =\> \["172.25.8.45:9200"\] index =\> \["heartbeat-\*"\] # query =\> '{ "query": { "match\_all": {} } }' # query =\> '{ "query": { "bool": {"must" : { "term" : { "monitor.status" : … --- ## [Split an event help needed](https://discuss.elastic.co/t/split-an-event-help-needed/181420)
**Author:** [@SathishPrakasam](https://discuss.elastic.co/u/SathishPrakasam)\ **Replies:** 4\ **Last updated:** [May 16, 2019, 4:59pm UTC](https://discuss.elastic.co/t/split-an-event-help-needed/181420 "2019-05-16T16:59:34Z")
Hi, i want help in split the field below, I want to create array of all the occurrence in the field below which starts with "START - PHASE 1 09:02:16.31, time is the dynamic value generated. Please find the input fi… --- ## [Logstash with IP-API](https://discuss.elastic.co/t/logstash-with-ip-api/181447)
**Author:** [@BenB196](https://discuss.elastic.co/u/BenB196)\ **Replies:** 1\ **Last updated:** [May 16, 2019, 4:23pm UTC](https://discuss.elastic.co/t/logstash-with-ip-api/181447 "2019-05-16T16:23:05Z")
Has anyone ever tried to integrate Logstash with http://ip-api.com. I know that Logstash has the GeoIP plugin which can be used. However, http://ip-api.com has a much cheaper and easier to maintain setup than having to u… --- ## [Parsing growing csv files with Logstash?](https://discuss.elastic.co/t/parsing-growing-csv-files-with-logstash/181442)
**Author:** [@sgreszcz](https://discuss.elastic.co/u/sgreszcz)\ **Replies:** 1\ **Last updated:** [May 16, 2019, 4:03pm UTC](https://discuss.elastic.co/t/parsing-growing-csv-files-with-logstash/181442 "2019-05-16T16:03:58Z")
I have a use case where I need to parse and index csv files that are periodically collected. The challenge is that the CSV file could grow and get overwritten with more rows for some time until the next CSV with a new f… --- ## [Logstash processing s3 bucket](https://discuss.elastic.co/t/logstash-processing-s3-bucket/181441)
**Author:** [@Maksim\_Sidorov](https://discuss.elastic.co/u/Maksim_Sidorov)\ **Replies:** 0\ **Last updated:** [May 16, 2019, 3:31pm UTC](https://discuss.elastic.co/t/logstash-processing-s3-bucket/181441 "2019-05-16T15:31:03Z")
Hi, I have problem with processing .gz file from AWS s3 bucket. If logstash try processing gz file with size 0. I get error. Error ERROR logstash.inputs.s3 - Gzip codec: We cannot uncompress the gzip file. maybe some… --- ## [How to do output filter](https://discuss.elastic.co/t/how-to-do-output-filter/181435)
**Author:** [@akhettal](https://discuss.elastic.co/u/akhettal)\ **Replies:** 0\ **Last updated:** [May 16, 2019, 3:15pm UTC](https://discuss.elastic.co/t/how-to-do-output-filter/181435 "2019-05-16T15:15:12Z")
Hi, Here is my output logstash.conf output { elasticsearch { hosts =\> \["127.0.0.1:9200"\] index =\> "logs-%{\[@metadata\]\[index\]}-v6-%{+YYYY.MM.dd}" manage\_template =\> false } if \[@type\]== "x" { file { pa… --- ## [CSV EPOCH to Timestamp](https://discuss.elastic.co/t/csv-epoch-to-timestamp/181273)
**Author:** [@teamg](https://discuss.elastic.co/u/teamg)\ **Replies:** 2\ **Last updated:** [May 16, 2019, 3:04pm UTC](https://discuss.elastic.co/t/csv-epoch-to-timestamp/181273 "2019-05-16T15:04:18Z")
I have a csv file with contents similar to this: UNIT\_ID,Serial\_Number,TIME UNIT4690,MMC3430089,1557866550 UNIT4691,MMC3500092,1557866545 UNIT4692,MMC5100135,1557866546 This first line are the column names. The T… --- ## [Logstash Filter Json To Array of Json](https://discuss.elastic.co/t/logstash-filter-json-to-array-of-json/181345)
**Author:** [@Vineethpp](https://discuss.elastic.co/u/Vineethpp)\ **Replies:** 7\ **Last updated:** [May 16, 2019, 2:52pm UTC](https://discuss.elastic.co/t/logstash-filter-json-to-array-of-json/181345 "2019-05-16T14:52:24Z")
Hi, I was trying get a logstash filter which will write json object to kafka as array of json object. As of now with out any filter its writing the json object, but for me i want the kafka events as array of json objec… --- ## [Can we monitore the connection between Logstash and ES](https://discuss.elastic.co/t/can-we-monitore-the-connection-between-logstash-and-es/181349)
**Author:** [@Rodolphe\_Ghio](https://discuss.elastic.co/u/Rodolphe_Ghio)\ **Replies:** 0\ **Last updated:** [May 16, 2019, 9:34am UTC](https://discuss.elastic.co/t/can-we-monitore-the-connection-between-logstash-and-es/181349 "2019-05-16T09:34:43Z")
Hi, I'm in a intership and I've to use for the first time ELK stack. I'm trying to solve a problem on my project. It seems me that Logstash and ES don't communicate. Cause I see anyone of my logs on kibana. So that's why… --- ## [Conditional pipeline](https://discuss.elastic.co/t/conditional-pipeline/181423)
**Author:** [@dkuiate1](https://discuss.elastic.co/u/dkuiate1)\ **Replies:** 1\ **Last updated:** [May 16, 2019, 2:43pm UTC](https://discuss.elastic.co/t/conditional-pipeline/181423 "2019-05-16T14:43:49Z")
Hello everyone, I want to apply a specific pipeline if a field exist in the pipeline.yml but i don't know how to figure it out. For exemple event with the field version1 will be sent to the pipeline 1 and event with th… --- ## [Backward regex replacement with gsub filter option](https://discuss.elastic.co/t/backward-regex-replacement-with-gsub-filter-option/181426)
**Author:** [@rahulkothanath](https://discuss.elastic.co/u/rahulkothanath)\ **Replies:** 1\ **Last updated:** [May 16, 2019, 2:37pm UTC](https://discuss.elastic.co/t/backward-regex-replacement-with-gsub-filter-option/181426 "2019-05-16T14:37:29Z")
I want to match a text and replace it with matched value using gsub regex. we ususally do it using $1 which is regex backward reference. the below code contain my requirement filter { mutate { gsub… --- ## [Logstash pipeline not finding generated object](https://discuss.elastic.co/t/logstash-pipeline-not-finding-generated-object/181302)
**Author:** [@Near](https://discuss.elastic.co/u/Near)\ **Replies:** 2\ **Last updated:** [May 16, 2019, 1:50pm UTC](https://discuss.elastic.co/t/logstash-pipeline-not-finding-generated-object/181302 "2019-05-16T13:50:07Z")
Hello! I am new to Logstash pipelining and I am trying my hand at some basic functionality before trying something more complex. HERE IS MY PROBLEM: I can't seem to access the fields that appear in the log message outpu… --- ## [Grok pattern for date](https://discuss.elastic.co/t/grok-pattern-for-date/181322)
**Author:** [@talial](https://discuss.elastic.co/u/talial)\ **Replies:** 5\ **Last updated:** [May 16, 2019, 1:19pm UTC](https://discuss.elastic.co/t/grok-pattern-for-date/181322 "2019-05-16T13:19:16Z")
Hi I have a log file that I want to parse to elasticsearch using logstash. In the log I have a date like that: 2019-04-18+05:48:54.470 I trying to find how to grok this timestamp with kibana Grok Debugger Trying the… --- ## [Date Filter recognized string format](https://discuss.elastic.co/t/date-filter-recognized-string-format/181382)
**Author:** [@antonio\_addivinola](https://discuss.elastic.co/u/antonio_addivinola)\ **Replies:** 2\ **Last updated:** [May 16, 2019, 1:18pm UTC](https://discuss.elastic.co/t/date-filter-recognized-string-format/181382 "2019-05-16T13:18:25Z")
Hi everyone! I have a problem using the logstash date filter. I have a date string formatted like this: ddMMyyyy'T'HHmmss in a field (called scheduledTime) of a json log. I need to convert this string into a date to … --- ## [Logstash 7.0.1 - Grok pattern](https://discuss.elastic.co/t/logstash-7-0-1-grok-pattern/181338)
**Author:** [@hassen\_k](https://discuss.elastic.co/u/hassen_k)\ **Replies:** 2\ **Last updated:** [May 16, 2019, 12:48pm UTC](https://discuss.elastic.co/t/logstash-7-0-1-grok-pattern/181338 "2019-05-16T12:48:51Z")
Hi, I am using Logstash 7.0.1. I have the following log : amPolicy:05/16/2019 05:47:31:262 AM CEST: Thread\[http-nio-8080-exec-14,5,main\] The Logstash configuration is like that : ... mutate { gsub =\> \[ "timestam… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=545) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=547)