# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=436 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 437 --- ## [If statement is failing to match for certain values.. pls help](https://discuss.elastic.co/t/if-statement-is-failing-to-match-for-certain-values-pls-help/210438)
**Author:** [@Cdnvballer](https://discuss.elastic.co/u/Cdnvballer)\ **Replies:** 3\ **Last updated:** [December 4, 2019, 9:24pm UTC](https://discuss.elastic.co/t/if-statement-is-failing-to-match-for-certain-values-pls-help/210438 "2019-12-04T21:24:54Z")
I have syslog network messages successfully parsed with logstash, event 1 has \[system\]\[syslog\]\[hostname\]=10.20.0.3 event 2 has \[system\]\[syslog\]\[hostname\]=10.111.0.3 however the following statements fail to match for 1… --- ## [Help with grok filter \[ipv4\]](https://discuss.elastic.co/t/help-with-grok-filter-ipv4/210426)
**Author:** [@raged](https://discuss.elastic.co/u/raged)\ **Replies:** 2\ **Last updated:** [December 4, 2019, 8:48pm UTC](https://discuss.elastic.co/t/help-with-grok-filter-ipv4/210426 "2019-12-04T20:48:28Z")
Afternoon all, We are utilizing metricbeat to send host info by using the metadata that is described in this article. It eventually gives me a field that is populated with IPs from all of the network adapters: "host":… --- ## [Pushing flush onto pipeline error without glob issue](https://discuss.elastic.co/t/pushing-flush-onto-pipeline-error-without-glob-issue/210255)
**Author:** [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)\ **Replies:** 1\ **Last updated:** [December 4, 2019, 6:34pm UTC](https://discuss.elastic.co/t/pushing-flush-onto-pipeline-error-without-glob-issue/210255 "2019-12-04T18:34:26Z")
I am having this output in logstash and unable to get new index created after the last time it worked last week. This doesnt look like an error but is definitely preventing logstash from creating index on kibana. Can any… --- ## [Is there a way to not explicitly set xpack.management.pipeline.id in centralized pipeline management?](https://discuss.elastic.co/t/is-there-a-way-to-not-explicitly-set-xpack-management-pipeline-id-in-centralized-pipeline-management/210579)
**Author:** [@seanziee](https://discuss.elastic.co/u/seanziee)\ **Replies:** 0\ **Last updated:** [December 4, 2019, 5:54pm UTC](https://discuss.elastic.co/t/is-there-a-way-to-not-explicitly-set-xpack-management-pipeline-id-in-centralized-pipeline-management/210579 "2019-12-04T17:54:05Z")
Hi, I'd like to use the central pipeline functionality to create logstash pipelines on the fly, but it seems like I need to pre determine the different ids of the configs I'd like to run in an array xpack.management.pi… --- ## [Creating an index from two others using logstash](https://discuss.elastic.co/t/creating-an-index-from-two-others-using-logstash/210567)
**Author:** [@Nsi](https://discuss.elastic.co/u/Nsi)\ **Replies:** 0\ **Last updated:** [December 4, 2019, 4:22pm UTC](https://discuss.elastic.co/t/creating-an-index-from-two-others-using-logstash/210567 "2019-12-04T16:22:42Z")
Hello, Being fairly new to the API, I am having a hard time trying to create an index containing some fields {a, b, c} from two other indexes containing some {a, b} and {b, c} fields respectively (there is a 'b' field o… --- ## [Can not use field from previous http filter](https://discuss.elastic.co/t/can-not-use-field-from-previous-http-filter/210533)
**Author:** [@Santiago](https://discuss.elastic.co/u/Santiago)\ **Replies:** 4\ **Last updated:** [December 4, 2019, 4:22pm UTC](https://discuss.elastic.co/t/can-not-use-field-from-previous-http-filter/210533 "2019-12-04T16:22:36Z")
Given the following config, I expect db\_size to have the same value as body.info.sizes.active. But it appears the copy can not resolve the nested field body.info.sizes.active. input { http\_poller { urls =\> { … --- ## [Pushing data to a kafka server](https://discuss.elastic.co/t/pushing-data-to-a-kafka-server/210553)
**Author:** [@Haalanam](https://discuss.elastic.co/u/Haalanam)\ **Replies:** 0\ **Last updated:** [December 4, 2019, 2:50pm UTC](https://discuss.elastic.co/t/pushing-data-to-a-kafka-server/210553 "2019-12-04T14:50:45Z")
AI am trying to load data into a kafka but it's not working somehow I am having this message when running my logstash conf \[DEBUG\]\[org.apache.kafka.clients.NetworkClient\]\[main\] \[Producer clientId=producer-1\] Sending me… --- ## [Logstash logs are read but are not pushed to elasticsearch](https://discuss.elastic.co/t/logstash-logs-are-read-but-are-not-pushed-to-elasticsearch/210319)
**Author:** [@ghimisradu](https://discuss.elastic.co/u/ghimisradu)\ **Replies:** 3\ **Last updated:** [December 4, 2019, 2:28pm UTC](https://discuss.elastic.co/t/logstash-logs-are-read-but-are-not-pushed-to-elasticsearch/210319 "2019-12-04T14:28:47Z")
I have the following logstash configuration: logstash-configration.confinput { file { codec =\> "json\_lines" path =\> \["/etc/logstash/input.log"\] sincedb\_path =\> "/etc/logstash/dbfil… --- ## [Google PubSub Not pulling new data](https://discuss.elastic.co/t/google-pubsub-not-pulling-new-data/209757)
**Author:** [@steevo590](https://discuss.elastic.co/u/steevo590)\ **Replies:** 1\ **Last updated:** [December 4, 2019, 2:11pm UTC](https://discuss.elastic.co/t/google-pubsub-not-pulling-new-data/209757 "2019-12-04T14:11:00Z")
Having an issue with the google pub/sub input plugin, I got it setup and it pulled in a bunch of records but hasn't/won't pull any new items down. Wondering if anyone has experience with this plugin or this behaviour. It… --- ## [Logstash pipeline got terminated after running a certain time but logstash is not getting shut down](https://discuss.elastic.co/t/logstash-pipeline-got-terminated-after-running-a-certain-time-but-logstash-is-not-getting-shut-down/210543)
**Author:** [@Avinaba\_Ghosh\_Hajra](https://discuss.elastic.co/u/Avinaba_Ghosh_Hajra)\ **Replies:** 0\ **Last updated:** [December 4, 2019, 1:42pm UTC](https://discuss.elastic.co/t/logstash-pipeline-got-terminated-after-running-a-certain-time-but-logstash-is-not-getting-shut-down/210543 "2019-12-04T13:42:04Z")
We are running a pipeline that will fetch the data from oracle db table and send it to elasticsearch. We are using jdbc input plugin for that. As it is a one time activity we are running it in the following way nohup lo… --- ## [Logstash Pipeline.yml - logstash.agent Failed to execute action](https://discuss.elastic.co/t/logstash-pipeline-yml-logstash-agent-failed-to-execute-action/210527)
**Author:** [@Apollyon](https://discuss.elastic.co/u/Apollyon)\ **Replies:** 0\ **Last updated:** [December 4, 2019, 12:03pm UTC](https://discuss.elastic.co/t/logstash-pipeline-yml-logstash-agent-failed-to-execute-action/210527 "2019-12-04T12:03:33Z")
Hello all, With Logstash 7.5.0 running, with us of the debug running a "LogStash::ConfigurationError" message appears saying that its expected "Expected one of \[ \\t\\r\\n\], "#", "input", "filter", "output" at line 1, colu… --- ## [Setting Logstash LS\_JAVA\_OPTS in docker not reflecting in container](https://discuss.elastic.co/t/setting-logstash-ls-java-opts-in-docker-not-reflecting-in-container/210187)
**Author:** [@jdronde](https://discuss.elastic.co/u/jdronde)\ **Replies:** 2\ **Last updated:** [December 4, 2019, 10:16am UTC](https://discuss.elastic.co/t/setting-logstash-ls-java-opts-in-docker-not-reflecting-in-container/210187 "2019-12-04T10:16:50Z")
Hi All, I hope that this is an easy one and something that i am doing wrong in my compose file to easily fix it. But basically in my compose file for logstash i am setting the environment value to LS\_JAVA\_OPTS logst… --- ## [Logstash bug in arcsight module?](https://discuss.elastic.co/t/logstash-bug-in-arcsight-module/210498)
**Author:** [@m4rc0](https://discuss.elastic.co/u/m4rc0)\ **Replies:** 0\ **Last updated:** [December 4, 2019, 9:47am UTC](https://discuss.elastic.co/t/logstash-bug-in-arcsight-module/210498 "2019-12-04T09:47:27Z")
Hi Team, we recently encounter strange WARN messages from Logstash since we're using 7.3.0 \[2019-12-04T08:46:18,174\]\[WARN \]\[org.apache.kafka.clients.NetworkClient\] \[Consumer clientId=logstash-0, groupId=logstash\] Conn… --- ## [Sincedb and ES unreachable](https://discuss.elastic.co/t/sincedb-and-es-unreachable/210491)
**Author:** [@boubakr](https://discuss.elastic.co/u/boubakr)\ **Replies:** 0\ **Last updated:** [December 4, 2019, 9:02am UTC](https://discuss.elastic.co/t/sincedb-and-es-unreachable/210491 "2019-12-04T09:02:13Z")
What happens to sincedb if I get this warning? \[WARN \]\[logstash.outputs.elasticsearch\] Attempted to resurrect connection to dead ES instance, but got an error. {:url=\>"http://localhost:9200/", :error\_type=\>LogStash::Ou… --- ## [Logstash 7.5.0 configuration error](https://discuss.elastic.co/t/logstash-7-5-0-configuration-error/210385)
**Author:** [@Jeffreyshoptrader](https://discuss.elastic.co/u/Jeffreyshoptrader)\ **Replies:** 2\ **Last updated:** [December 4, 2019, 8:25am UTC](https://discuss.elastic.co/t/logstash-7-5-0-configuration-error/210385 "2019-12-04T08:25:43Z")
Hi I'm new to ELK and I want to connect Logstash to Elasticsearch, but my data doesn't go to Elasticsearch. Everytime I try to run Logstash I get an error. \[2019-12-03T16:14:10,102\]\[INFO \]\[logstash.runner \] Sta… --- ## [Logstash Configure](https://discuss.elastic.co/t/logstash-configure/210471)
**Author:** [@Vo\_Phan\_Hong\_Dung](https://discuss.elastic.co/u/Vo_Phan_Hong_Dung)\ **Replies:** 0\ **Last updated:** [December 4, 2019, 6:44am UTC](https://discuss.elastic.co/t/logstash-configure/210471 "2019-12-04T06:44:35Z")
Hello guys, Please help me create a config logstash to sync data from sql server to elastic-search. Thanks --- ## [Grok filter for the logs coming from Nagios](https://discuss.elastic.co/t/grok-filter-for-the-logs-coming-from-nagios/210435)
**Author:** [@Morymogh](https://discuss.elastic.co/u/Morymogh)\ **Replies:** 1\ **Last updated:** [December 3, 2019, 11:07pm UTC](https://discuss.elastic.co/t/grok-filter-for-the-logs-coming-from-nagios/210435 "2019-12-03T23:07:46Z")
Can someone please assist to write a grok filter for the below raw data captured by tcpdump? The logs are tagged as "siem" to be forwarded from a Nagios server to our elasticsearch, basically \["siem"\] shows end of the e… --- ## [Output of \`logstash -e 'input { stdin { } } output { stdout {} }'\`](https://discuss.elastic.co/t/output-of-logstash-e-input-stdin-output-stdout/210244)
**Author:** [@Willl](https://discuss.elastic.co/u/Willl)\ **Replies:** 2\ **Last updated:** [December 3, 2019, 9:24pm UTC](https://discuss.elastic.co/t/output-of-logstash-e-input-stdin-output-stdout/210244 "2019-12-03T21:24:08Z")
Hi Testing a new 7.5.0 setup. Brand new. Logstash doc says test it with logstash -e 'input { stdin { } } output { stdout {} }' and then enter "hello world" after logstash says "pipeline is started" and I should se… --- ## [Set a template\_name in Logstash output filter to send data elasticsearch](https://discuss.elastic.co/t/set-a-template-name-in-logstash-output-filter-to-send-data-elasticsearch/210233)
**Author:** [@aveek](https://discuss.elastic.co/u/aveek)\ **Replies:** 2\ **Last updated:** [December 3, 2019, 6:00pm UTC](https://discuss.elastic.co/t/set-a-template-name-in-logstash-output-filter-to-send-data-elasticsearch/210233 "2019-12-03T18:00:44Z")
Hi Guys, We am using Elastic Stack 7.2.0 . We have setup filebeat to ship data directly to elasticsearch but we have some application log files which requires special parsing. We are sending these logs from filebeat to … --- ## [Logstash Date Parsing Error](https://discuss.elastic.co/t/logstash-date-parsing-error/210409)
**Author:** [@Triack\_2000](https://discuss.elastic.co/u/Triack_2000)\ **Replies:** 0\ **Last updated:** [December 3, 2019, 5:40pm UTC](https://discuss.elastic.co/t/logstash-date-parsing-error/210409 "2019-12-03T17:40:16Z")
Hello All, I am trying to parse logs from openresty and running into parsing issue. In the log, the date shows as: \[03/Dec/2019:17:17:14 +0000\] my parser is configured as: date{ match =\> \[ "log\_time", "dd/MMM/yyyy:HH:… --- ## [Logstash output - All messages outputing to one message field](https://discuss.elastic.co/t/logstash-output-all-messages-outputing-to-one-message-field/210399)
**Author:** [@ksarpong](https://discuss.elastic.co/u/ksarpong)\ **Replies:** 1\ **Last updated:** [December 3, 2019, 5:02pm UTC](https://discuss.elastic.co/t/logstash-output-all-messages-outputing-to-one-message-field/210399 "2019-12-03T17:02:19Z")
I am using a standard out with ruby for out, no codec defined for input and all messages are being send in on message field. The vendor I am working with is saying we need to get a codec that can seperate messages by ne… --- ## [Logstash unable to process pipelines.yml](https://discuss.elastic.co/t/logstash-unable-to-process-pipelines-yml/210390)
**Author:** [@BeMoore](https://discuss.elastic.co/u/BeMoore)\ **Replies:** 0\ **Last updated:** [December 3, 2019, 3:44pm UTC](https://discuss.elastic.co/t/logstash-unable-to-process-pipelines-yml/210390 "2019-12-03T15:44:29Z")
So i'm wrestling with a new problem today... Logstash 7.5.0 recently upgraded my test ELK stack from 7.4.2 to 7.5.0 Immediately hit an issue ingestion data with my setup. the error i get is Dec 03 15:14:49 ELK01 logs… --- ## [Pipelines.yml keeps showing error in debug](https://discuss.elastic.co/t/pipelines-yml-keeps-showing-error-in-debug/210389)
**Author:** [@Apollyon](https://discuss.elastic.co/u/Apollyon)\ **Replies:** 0\ **Last updated:** [December 3, 2019, 3:33pm UTC](https://discuss.elastic.co/t/pipelines-yml-keeps-showing-error-in-debug/210389 "2019-12-03T15:33:59Z")
Hello, After upgrading to 7.5.0, when trying to set Logstash up again, errors keep occurring relating to the "Failed to execute". Everything was working before the upgrade, but now it doesn't seem to like what is there.… --- ## [Using local logstash to connect to AWS Elastic](https://discuss.elastic.co/t/using-local-logstash-to-connect-to-aws-elastic/210359)
**Author:** [@soak](https://discuss.elastic.co/u/soak)\ **Replies:** 0\ **Last updated:** [December 3, 2019, 1:40pm UTC](https://discuss.elastic.co/t/using-local-logstash-to-connect-to-aws-elastic/210359 "2019-12-03T13:40:21Z")
Followed these instructions https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-vpc.html#es-vpc-security I get the following error running the logstash command \[ERROR\]\[logstash.javapipeline \]… --- ## [\[logstash.config.source.multilocal\] Ignoring the 'pipelines.yml' file because modules or command line options are specified](https://discuss.elastic.co/t/logstash-config-source-multilocal-ignoring-the-pipelines-yml-file-because-modules-or-command-line-options-are-specified/210343)
**Author:** [@sampath1](https://discuss.elastic.co/u/sampath1)\ **Replies:** 1\ **Last updated:** [December 3, 2019, 1:11pm UTC](https://discuss.elastic.co/t/logstash-config-source-multilocal-ignoring-the-pipelines-yml-file-because-modules-or-command-line-options-are-specified/210343 "2019-12-03T13:11:36Z")
Hi, I am facing an issue. can anyone help me out this. config.code: input { jdbc { jdbc\_connection\_string =\> "jdbc:mysql://192.168.10.111:3306/MTv5" # The user we wish to execute our statement as jdbc\_user =\> "rea… --- ## [Logstash converting date / Time in correctly](https://discuss.elastic.co/t/logstash-converting-date-time-in-correctly/210266)
**Author:** [@Trevorp](https://discuss.elastic.co/u/Trevorp)\ **Replies:** 1\ **Last updated:** [December 3, 2019, 1:05pm UTC](https://discuss.elastic.co/t/logstash-converting-date-time-in-correctly/210266 "2019-12-03T13:05:46Z")
When I am importing data to logstash it does not handle the data/Time correctly It shows time Dec 2, 2019 @ 21:09:29.000 when it should show 2019/12/03 05:09:29 Message in Kibana tmessage 2019/12/03 05:09:29,95,MP M… --- ## [Elastic Stack QRadar Integration](https://discuss.elastic.co/t/elastic-stack-qradar-integration/210247)
**Author:** [@Booooooo](https://discuss.elastic.co/u/Booooooo)\ **Replies:** 1\ **Last updated:** [December 3, 2019, 1:03pm UTC](https://discuss.elastic.co/t/elastic-stack-qradar-integration/210247 "2019-12-03T13:03:25Z")
Hi, So I have been playing with forwarding events from QRadar to the Elastic Stack. I have configured QRadar to forward events to Logstash via TCP in JSON format. On Logstash I have... Input: input { tcp { … --- ## [Replace the value of a field](https://discuss.elastic.co/t/replace-the-value-of-a-field/210171)
**Author:** [@wedkarz014](https://discuss.elastic.co/u/wedkarz014)\ **Replies:** 2\ **Last updated:** [December 3, 2019, 7:36am UTC](https://discuss.elastic.co/t/replace-the-value-of-a-field/210171 "2019-12-03T07:36:21Z")
Hi, I'm trying to add a few characters at the beginning and the end of a field but i have an error. This is what i'd like to add " { "info": " and "}" I tried: mutate { replace =\> { "message" =\> "{\\"inf… --- ## [How to remove full\_message or message from log under filter](https://discuss.elastic.co/t/how-to-remove-full-message-or-message-from-log-under-filter/210289)
**Author:** [@tharu85](https://discuss.elastic.co/u/tharu85)\ **Replies:** 4\ **Last updated:** [December 3, 2019, 5:51am UTC](https://discuss.elastic.co/t/how-to-remove-full-message-or-message-from-log-under-filter/210289 "2019-12-03T05:51:30Z")
I need to remove full\_message or message field using logstash filter. Keep both these values increase the disk capacity. I tried following syntax under logstash filter. filter { grok { remove\_field =\> \["full\_mess… --- ## [Convert object into nested array using Ruby Logstash](https://discuss.elastic.co/t/convert-object-into-nested-array-using-ruby-logstash/210160)
**Author:** [@Reshmi](https://discuss.elastic.co/u/Reshmi)\ **Replies:** 2\ **Last updated:** [December 3, 2019, 5:41am UTC](https://discuss.elastic.co/t/convert-object-into-nested-array-using-ruby-logstash/210160 "2019-12-03T05:41:40Z")
My XML structure is \ \ \ After xml parsing with the below code xml{ source =\> "message" store\_xml =\> true target =\> … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=435) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=437)