# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=609 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 610 --- ## [Get the XML node from unformatted xml](https://discuss.elastic.co/t/get-the-xml-node-from-unformatted-xml/163576)
**Author:** [@rainman](https://discuss.elastic.co/u/rainman)\ **Replies:** 8\ **Last updated:** [January 30, 2019, 3:41pm UTC](https://discuss.elastic.co/t/get-the-xml-node-from-unformatted-xml/163576 "2019-01-30T15:41:49Z")
I have a xml that is not parsing correctly unless I prettify the xml. XML tags can be a new line or same line with the last end tag, and does not have indentation for it. but i tried with prettify it worked fine. can we … --- ## [Add\_field getting wrong value](https://discuss.elastic.co/t/add-field-getting-wrong-value/166406)
**Author:** [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Replies:** 4\ **Last updated:** [January 30, 2019, 6:27pm UTC](https://discuss.elastic.co/t/add-field-getting-wrong-value/166406 "2019-01-30T18:27:39Z")
I am using jdbc to connect to database getting different field trying to make new field but it give me something weired output. what am I doing wrong? filter { mutate { add\_field =\> { "did" =\> "%{serial}%{sid}%{uid}… --- ## [How to connect jhipster on aws to elk cloud](https://discuss.elastic.co/t/how-to-connect-jhipster-on-aws-to-elk-cloud/166410)
**Author:** [@Mario\_Aliaga](https://discuss.elastic.co/u/Mario_Aliaga)\ **Replies:** 0\ **Last updated:** [January 30, 2019, 5:52pm UTC](https://discuss.elastic.co/t/how-to-connect-jhipster-on-aws-to-elk-cloud/166410 "2019-01-30T17:52:24Z")
Hello everyone, I need to connect my app that is in AWS / elasticbeanstalk to ELK Cloud. It does not work with this: jhipster: logging: logstash: enabled: true host: localhost # … --- ## [Logstash wildcards / regex not working](https://discuss.elastic.co/t/logstash-wildcards-regex-not-working/164204)
**Author:** [@Blake\_2112](https://discuss.elastic.co/u/Blake_2112)\ **Replies:** 20\ **Last updated:** [January 30, 2019, 5:31pm UTC](https://discuss.elastic.co/t/logstash-wildcards-regex-not-working/164204 "2019-01-30T17:31:03Z")
No matter what I have tried I cannot seem to easily process files in a folder using a repeatable wildcard format. To start...there are no errors...just no processing as I am certain that logstash cannot find any files a… --- ## [How to fix this error](https://discuss.elastic.co/t/how-to-fix-this-error/166386)
**Author:** [@AmritMatti](https://discuss.elastic.co/u/AmritMatti)\ **Replies:** 4\ **Last updated:** [January 30, 2019, 5:02pm UTC](https://discuss.elastic.co/t/how-to-fix-this-error/166386 "2019-01-30T17:02:28Z")
2019-01-30T15:08:30.215Z ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://104.43.129.213:5044)): dial tcp 104.43.129.213:5044: connect: connection refused 2019-01-30T15:08:30.216Z I… --- ## [Get json output from xml file input in logstash](https://discuss.elastic.co/t/get-json-output-from-xml-file-input-in-logstash/166347)
**Author:** [@manasapp](https://discuss.elastic.co/u/manasapp)\ **Replies:** 7\ **Last updated:** [January 30, 2019, 3:52pm UTC](https://discuss.elastic.co/t/get-json-output-from-xml-file-input-in-logstash/166347 "2019-01-30T15:52:28Z")
Hi Everyone, i want to convert xml input to json data. The xml format is shown below \ \ \
\ **Author:** [@fooc](https://discuss.elastic.co/u/fooc)\ **Replies:** 4\ **Last updated:** [January 30, 2019, 3:36pm UTC](https://discuss.elastic.co/t/grokparsefailure-geoip-lookup-failure-sophos-utm/166167 "2019-01-30T15:36:00Z") Hi, I'm using the following conf file to process my syslog messages from a sophos utm: input { tcp { type =\> "utm-syslog" port =\> 5140 } } filter { if \[type\] == "utm-syslog" { grok { match =\> { "message" =\> "\<%{… --- ## [KV filter: Detect malformed messages](https://discuss.elastic.co/t/kv-filter-detect-malformed-messages/166266)
**Author:** [@imehl](https://discuss.elastic.co/u/imehl)\ **Replies:** 1\ **Last updated:** [January 30, 2019, 2:48pm UTC](https://discuss.elastic.co/t/kv-filter-detect-malformed-messages/166266 "2019-01-30T14:48:47Z")
Hi, I have incoming messages with the format: key1=5 key2='value 2' key3=value (many values, dynamic) ...and I'm using the KV filter to extract the key value pairs, which works as expected. Now I also have incoming … --- ## [Logstash configuration file error filtering](https://discuss.elastic.co/t/logstash-configuration-file-error-filtering/166338)
**Author:** [@althair34](https://discuss.elastic.co/u/althair34)\ **Replies:** 5\ **Last updated:** [January 30, 2019, 1:41pm UTC](https://discuss.elastic.co/t/logstash-configuration-file-error-filtering/166338 "2019-01-30T13:41:22Z")
Hello, I'm starting to use ELK and I'm attempting to start logstash with a custom logstash.conf file but I've an error : Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>… --- ## [How does Logstash parsing the below events into correct json data?](https://discuss.elastic.co/t/how-does-logstash-parsing-the-below-events-into-correct-json-data/165985)
**Author:** [@Robin\_Guo](https://discuss.elastic.co/u/Robin_Guo)\ **Replies:** 3\ **Last updated:** [January 30, 2019, 1:37pm UTC](https://discuss.elastic.co/t/how-does-logstash-parsing-the-below-events-into-correct-json-data/165985 "2019-01-30T13:37:32Z")
Note: the raw PHP larval log is consist of JSON object format text and non-JSON format text Original PHP larval log: \[2019-01-28 02:28:22\] prod.INFO: {"database":"test10","type":"form","id":"20821","comment":"passba… --- ## [Jmx-input-plugin supports jmx authentication](https://discuss.elastic.co/t/jmx-input-plugin-supports-jmx-authentication/166356)
**Author:** [@azam](https://discuss.elastic.co/u/azam)\ **Replies:** 1\ **Last updated:** [January 30, 2019, 1:23pm UTC](https://discuss.elastic.co/t/jmx-input-plugin-supports-jmx-authentication/166356 "2019-01-30T13:23:38Z")
Hey all We are trying to ship JMX data/metrics to our logstash. The problem is, our JMX uses authentication which is necessary to secure our applications. I would like to know, does jmx-input plugin supports authenticat… --- ## [Logstash deadletterqueue](https://discuss.elastic.co/t/logstash-deadletterqueue/166195)
**Author:** [@fadihaddad](https://discuss.elastic.co/u/fadihaddad)\ **Replies:** 3\ **Last updated:** [January 30, 2019, 1:13pm UTC](https://discuss.elastic.co/t/logstash-deadletterqueue/166195 "2019-01-30T13:13:21Z")
i enabled deadletterqueue in logtsash.yml and created the pipeline deadletterqueue in pipelines.yml wih id dlq so this os my config input { dead\_letter\_queue { path =\> "/var/lib/logstash/dead\_letter\_queue" commit\_of… --- ## [Configure logstash for dev and prod with different index pattern](https://discuss.elastic.co/t/configure-logstash-for-dev-and-prod-with-different-index-pattern/165853)
**Author:** [@sundarm](https://discuss.elastic.co/u/sundarm)\ **Replies:** 5\ **Last updated:** [January 30, 2019, 11:31am UTC](https://discuss.elastic.co/t/configure-logstash-for-dev-and-prod-with-different-index-pattern/165853 "2019-01-30T11:31:08Z")
Hi , I am having single logstash server and installed filebeats on dev and prod instances and in my logstash config i am using the following index pattern index =\> "applogs-%{+YYYY.MM.dd}" so now all my dev and prod … --- ## [Create a new field with ruby filter](https://discuss.elastic.co/t/create-a-new-field-with-ruby-filter/166307)
**Author:** [@thelfter](https://discuss.elastic.co/u/thelfter)\ **Replies:** 1\ **Last updated:** [January 30, 2019, 11:00am UTC](https://discuss.elastic.co/t/create-a-new-field-with-ruby-filter/166307 "2019-01-30T11:00:43Z")
grok { match =\> { "request" =\> \[ "url", "%{URIPATH:url\_path}%{URIPARAM:url\_params}?" \]} } urldecode{ field =\> "url\_path" } mutate { gsub =\> \["url\_params","\\?","" \] } kv { field\_split =\> "&" source =\> "url\_… --- ## [Solution to get Account actually looked out in AD](https://discuss.elastic.co/t/solution-to-get-account-actually-looked-out-in-ad/166332)
**Author:** [@Thibaut603](https://discuss.elastic.co/u/Thibaut603)\ **Replies:** 1\ **Last updated:** [January 30, 2019, 10:37am UTC](https://discuss.elastic.co/t/solution-to-get-account-actually-looked-out-in-ad/166332 "2019-01-30T10:37:58Z")
Hello, i'm student and for my final test i need to implemant elk (elastic, kibana, logstash, winlogbeat ) And i want to get all accounts who are curently looked out. I things the solution is a script on AD to get this… --- ## [Nested json output to csv by using conditional statements in logstash](https://discuss.elastic.co/t/nested-json-output-to-csv-by-using-conditional-statements-in-logstash/165719)
**Author:** [@manasapp](https://discuss.elastic.co/u/manasapp)\ **Replies:** 4\ **Last updated:** [January 30, 2019, 9:39am UTC](https://discuss.elastic.co/t/nested-json-output-to-csv-by-using-conditional-statements-in-logstash/165719 "2019-01-30T09:39:42Z")
hello guys, iam using metric beat as input to log stash and i got output as shown below {"metricset":{"module":"system","name":"memory","rtt":822},"@timestamp":"2019-01-22T18:45:38.755Z","beat":{"version":"6.5.4","name… --- ## [Replace @timestamp with timestamp in my message - logstash](https://discuss.elastic.co/t/replace-timestamp-with-timestamp-in-my-message-logstash/166000)
**Author:** [@manasapp](https://discuss.elastic.co/u/manasapp)\ **Replies:** 11\ **Last updated:** [January 30, 2019, 9:39am UTC](https://discuss.elastic.co/t/replace-timestamp-with-timestamp-in-my-message-logstash/166000 "2019-01-30T09:39:02Z")
Hi Everyone i need to replace @timestamp with the timestamp from log file(message). still i can't mention complete message format. i can use like starts with timestamp. i tried below code grok { match =\> { "message"… --- ## [Get message content without message tag in logstash](https://discuss.elastic.co/t/get-message-content-without-message-tag-in-logstash/165714)
**Author:** [@manasapp](https://discuss.elastic.co/u/manasapp)\ **Replies:** 10\ **Last updated:** [January 30, 2019, 9:35am UTC](https://discuss.elastic.co/t/get-message-content-without-message-tag-in-logstash/165714 "2019-01-30T09:35:00Z")
Hello everyone, This is the config file i used. input { syslog { port =\> 514 } } filter{ mutate { remove\_field =\> \["facility\_label","facility","@version","priority","tags","host","timestamp","severity\_label"… --- ## [Files limit](https://discuss.elastic.co/t/files-limit/166254)
**Author:** [@niraj\_pandey](https://discuss.elastic.co/u/niraj_pandey)\ **Replies:** 1\ **Last updated:** [January 30, 2019, 5:29am UTC](https://discuss.elastic.co/t/files-limit/166254 "2019-01-30T05:29:40Z")
Hello, On logstash log file I am continuously getting the following warning messages. Could you please help me on this ? \[2019-01-29T20:03:51,022\]\[WARN \]\[filewatch.tailmode.processor\] Reached open files limit: 4095, s… --- ## [\[Basic Question\] - KV Filter avoid using a separator on a field](https://discuss.elastic.co/t/basic-question-kv-filter-avoid-using-a-separator-on-a-field/165680)
**Author:** [@Rh1n0](https://discuss.elastic.co/u/Rh1n0)\ **Replies:** 4\ **Last updated:** [January 29, 2019, 9:48pm UTC](https://discuss.elastic.co/t/basic-question-kv-filter-avoid-using-a-separator-on-a-field/165680 "2019-01-29T21:48:52Z")
Guys hello, i'm very new to all the ELK stack. Right now i'm trying to parse some emailing logs that have an structure simillar to this: msgid=XXXXXX, orig\_msgid='\', relay=(), status='Email Deferred', rece… --- ## [How to use VPC endpoints for Amazon SQS (AWS PrivateLink) in SQS input plugin](https://discuss.elastic.co/t/how-to-use-vpc-endpoints-for-amazon-sqs-aws-privatelink-in-sqs-input-plugin/166236)
**Author:** [@Jcneisius](https://discuss.elastic.co/u/Jcneisius)\ **Replies:** 0\ **Last updated:** [January 29, 2019, 9:17pm UTC](https://discuss.elastic.co/t/how-to-use-vpc-endpoints-for-amazon-sqs-aws-privatelink-in-sqs-input-plugin/166236 "2019-01-29T21:17:38Z")
Hi To minimise run cost in AWS, I would like to fetch SQS messages from logstash sqs input plugin without requiring an internet gateway. AWS customers can now access Amazon SQS using VPC endpoints, without using public… --- ## [Logstash RabbitMQ input plugin configuration](https://discuss.elastic.co/t/logstash-rabbitmq-input-plugin-configuration/166222)
**Author:** [@shimpi](https://discuss.elastic.co/u/shimpi)\ **Replies:** 5\ **Last updated:** [January 29, 2019, 8:57pm UTC](https://discuss.elastic.co/t/logstash-rabbitmq-input-plugin-configuration/166222 "2019-01-29T20:57:45Z")
Hello, My goal is: 1: Configure Logstash (version: 6.2.1) to pull events from RabbitMQ (version: 3.6.1) ( as input) 2: Depending upon the content of the event (message), store them in different files (as output) Righ… --- ## [Logstash not reading from file input](https://discuss.elastic.co/t/logstash-not-reading-from-file-input/166180)
**Author:** [@jbucklew](https://discuss.elastic.co/u/jbucklew)\ **Replies:** 16\ **Last updated:** [January 29, 2019, 7:40pm UTC](https://discuss.elastic.co/t/logstash-not-reading-from-file-input/166180 "2019-01-29T19:40:48Z")
I'm running this implementation of the ELK stack, which is pretty straightforward and easy to configure. I can push TCP input through the stack using netcat like so: nc localhost 5000 \< /Users/me/path/to/logs/appOne.lo… --- ## [Obtener un json de una entrada de log](https://discuss.elastic.co/t/obtener-un-json-de-una-entrada-de-log/166217)
**Author:** [@Lucas\_Fernandez](https://discuss.elastic.co/u/Lucas_Fernandez)\ **Replies:** 8\ **Last updated:** [January 29, 2019, 6:43pm UTC](https://discuss.elastic.co/t/obtener-un-json-de-una-entrada-de-log/166217 "2019-01-29T18:43:57Z")
Buenas!! necesitaria poder obtener un json de una entrada de log y poder convertir los campos de ese json a campos queryables (cabe destacar que esos campos pueden estar desordenados) en kibana a traves de elasticsearch. … --- ## [Optimize logstash filter plugin with million lines of dictionary look up](https://discuss.elastic.co/t/optimize-logstash-filter-plugin-with-million-lines-of-dictionary-look-up/164723)
**Author:** [@ronchav](https://discuss.elastic.co/u/ronchav)\ **Replies:** 15\ **Last updated:** [January 29, 2019, 6:05pm UTC](https://discuss.elastic.co/t/optimize-logstash-filter-plugin-with-million-lines-of-dictionary-look-up/164723 "2019-01-29T18:05:51Z")
Hi, I am implementing data masking which is based in a dictionary lookup. Currently there are four dictionary files (total of ~1.2 million lines) reference to translate my greedy message. When transformation runs using… --- ## [Stop logsstash](https://discuss.elastic.co/t/stop-logsstash/165279)
**Author:** [@gongon](https://discuss.elastic.co/u/gongon)\ **Replies:** 14\ **Last updated:** [January 29, 2019, 5:30pm UTC](https://discuss.elastic.co/t/stop-logsstash/165279 "2019-01-29T17:30:31Z")
Hello, I can't stop Logstash so I can't test any config file on my mac. Here the error : I3893-2:6.4.1 gongon$ logstash -f first-pipeline.conf Sending Logstash logs to /usr/local/Cellar/logstash/6.4.1/libexec/logs whic… --- ## [Get exiting in ES index documents to Logstash pipeline to make calculations](https://discuss.elastic.co/t/get-exiting-in-es-index-documents-to-logstash-pipeline-to-make-calculations/165050)
**Author:** [@dfesenko](https://discuss.elastic.co/u/dfesenko)\ **Replies:** 2\ **Last updated:** [January 29, 2019, 2:37pm UTC](https://discuss.elastic.co/t/get-exiting-in-es-index-documents-to-logstash-pipeline-to-make-calculations/165050 "2019-01-29T14:37:30Z")
Hello I need to create documents in ES index based on the unique IDs of the events. And when the ID of the new event will be the same as the ID of the documents already stored in the ES index, I want NOT TO overwrite it… --- ## [Logstash on cloud](https://discuss.elastic.co/t/logstash-on-cloud/163339)
**Author:** [@eebee](https://discuss.elastic.co/u/eebee)\ **Replies:** 2\ **Last updated:** [January 29, 2019, 2:00pm UTC](https://discuss.elastic.co/t/logstash-on-cloud/163339 "2019-01-29T14:00:54Z")
I'm trying out the free trial of Elasticsearch on Cloud. I'm trying to add a Logstash Pipeline to ingest data from an external source. How do I configure the output-part to ingest my data? output { elasticsearch { h… --- ## [How obtain only a part of the log?](https://discuss.elastic.co/t/how-obtain-only-a-part-of-the-log/166042)
**Author:** [@Lucas\_Fernandez](https://discuss.elastic.co/u/Lucas_Fernandez)\ **Replies:** 2\ **Last updated:** [January 29, 2019, 1:36pm UTC](https://discuss.elastic.co/t/how-obtain-only-a-part-of-the-log/166042 "2019-01-29T13:36:41Z")
Hello! I only want know, how to get only a part of log string. My input is from filebeat and my output is elasticseach base. I have this logstash.conf input { beats { port =\> 5044 } } filter { grok { match =\> { … --- ## [Elapsed filter. Random messages order in pipeline](https://discuss.elastic.co/t/elapsed-filter-random-messages-order-in-pipeline/166158)
**Author:** [@bodi4](https://discuss.elastic.co/u/bodi4)\ **Replies:** 1\ **Last updated:** [January 29, 2019, 1:29pm UTC](https://discuss.elastic.co/t/elapsed-filter-random-messages-order-in-pipeline/166158 "2019-01-29T13:29:07Z")
Bumped the problem with Elapsed filter. Using single pipeline worker but still... Trying to calculate elapsed time between 2 events from different log files on same host. Both events have uniqe field, correct tags, sh… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=608) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=610)