# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=493 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 494 --- ## [Retrieve data from API, split response array and handle duplicates in Logstash](https://discuss.elastic.co/t/retrieve-data-from-api-split-response-array-and-handle-duplicates-in-logstash/195225)
**Author:** [@frv](https://discuss.elastic.co/u/frv)\ **Replies:** 0\ **Last updated:** [August 14, 2019, 3:09pm UTC](https://discuss.elastic.co/t/retrieve-data-from-api-split-response-array-and-handle-duplicates-in-logstash/195225 "2019-08-14T15:09:21Z")
I am working on an application that pulls in news articles from an API using the http\_poller plugin for Logstash. I want the data to be searchable in Elasticsearch. The basic setup that I have so far is working properly… --- ## [Grok Pattern not matching in Logstash, but matches in Grok Debuggger](https://discuss.elastic.co/t/grok-pattern-not-matching-in-logstash-but-matches-in-grok-debuggger/194985)
**Author:** [@man0l](https://discuss.elastic.co/u/man0l)\ **Replies:** 8\ **Last updated:** [August 14, 2019, 2:52pm UTC](https://discuss.elastic.co/t/grok-pattern-not-matching-in-logstash-but-matches-in-grok-debuggger/194985 "2019-08-14T14:52:17Z")
Hi, I have this grok pattern: (?\%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME})%{SPACE}%{LOGLEVEL:level}%{SPACE}%{NUMBER:pid} --- \\\[%{SPACE}(?\\[A-Za-z0-9-\]+)\\\] \[A-Za-z0-9.\]\*\\.(?\\[A-Za-z0-9#\_\]+)\\s\*:%{S… --- ## [Logstash network output blocking - 7.3.0](https://discuss.elastic.co/t/logstash-network-output-blocking-7-3-0/195037)
**Author:** [@espenk](https://discuss.elastic.co/u/espenk)\ **Replies:** 3\ **Last updated:** [August 14, 2019, 2:14pm UTC](https://discuss.elastic.co/t/logstash-network-output-blocking-7-3-0/195037 "2019-08-14T14:14:58Z")
Hi. I'm currently having issues with a Logstash instance not being utilized properly. Maximum load is ~ 50% on a 18 core VM running CentOS 7.6.1810. If I replace tcp output{} with file output{} , the load increase to ~90… --- ## [S3 input plugin. Parse again an S3 bucket](https://discuss.elastic.co/t/s3-input-plugin-parse-again-an-s3-bucket/194397)
**Author:** [@Adrien\_Bigot](https://discuss.elastic.co/u/Adrien_Bigot)\ **Replies:** 2\ **Last updated:** [August 14, 2019, 1:15pm UTC](https://discuss.elastic.co/t/s3-input-plugin-parse-again-an-s3-bucket/194397 "2019-08-14T13:15:03Z")
Hello, I have an S3 bucket filled with kinesis logs. I want to re scan entirely this bucket but I didn't manage to do it. I've tried to delete the sincedb file ( /var/lib/logstash/plugins/inputs/s3/sincedb\_xxxxxxxxxxx… --- ## [Logstash fingerprint with concatenate\_all\_fields validation](https://discuss.elastic.co/t/logstash-fingerprint-with-concatenate-all-fields-validation/195193)
**Author:** [@PA3mEP](https://discuss.elastic.co/u/PA3mEP)\ **Replies:** 1\ **Last updated:** [August 14, 2019, 12:40pm UTC](https://discuss.elastic.co/t/logstash-fingerprint-with-concatenate-all-fields-validation/195193 "2019-08-14T12:40:49Z")
Hi, I need some help with the fingerprint validation that was produced using fingerprint logstash filter. at the moment pipeline config looks like this: input { jdbc { jdbc\_driver\_library =\> "/usr/share/logstash/log… --- ## [Count on no of log lines per month](https://discuss.elastic.co/t/count-on-no-of-log-lines-per-month/195200)
**Author:** [@Eva](https://discuss.elastic.co/u/Eva)\ **Replies:** 0\ **Last updated:** [August 14, 2019, 12:21pm UTC](https://discuss.elastic.co/t/count-on-no-of-log-lines-per-month/195200 "2019-08-14T12:21:25Z")
Hello Are there better ways to find number of log lines that are getting accumulated in elk on monthly basis? Thanks --- ## [Azure event hub failing with 'undefined method \`getEventHubPath' error](https://discuss.elastic.co/t/azure-event-hub-failing-with-undefined-method-geteventhubpath-error/195198)
**Author:** [@asad\_ali](https://discuss.elastic.co/u/asad_ali)\ **Replies:** 0\ **Last updated:** [August 14, 2019, 12:15pm UTC](https://discuss.elastic.co/t/azure-event-hub-failing-with-undefined-method-geteventhubpath-error/195198 "2019-08-14T12:15:18Z")
Hi, My logstash config looks like input { azure\_event\_hubs { config\_mode =\> "basic" event\_hub\_connections =\> \["Endpoint=sb://audit-ops-logs.servicebus.windows.net/;SharedAccessKeyName=azure-ls;SharedAccessKey=xxxxx… --- ## [Host name incorrect for logstash monitoring in Kibana](https://discuss.elastic.co/t/host-name-incorrect-for-logstash-monitoring-in-kibana/194957)
**Author:** [@Gokul\_Raj](https://discuss.elastic.co/u/Gokul_Raj)\ **Replies:** 1\ **Last updated:** [August 14, 2019, 9:46am UTC](https://discuss.elastic.co/t/host-name-incorrect-for-logstash-monitoring-in-kibana/194957 "2019-08-14T09:46:34Z")
I was able to configure logstash 6.7.1 to enable monitoring and it's successfully showing up in Kibana. However the host name is incorrect, it defaults to 127.0.0.1:9600 and 127.0.0.1:9601 for all 3 nodes in my cluster.… --- ## [LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError: Got response code '403' contacting Elasticsearch at URL 'http://localhost:9200/logstash'](https://discuss.elastic.co/t/logstash-got-response-code-403-contacting-elasticsearch-at-url-http-localhost-9200-logstash/194951)
**Author:** [@Daniel6](https://discuss.elastic.co/u/Daniel6)\ **Replies:** 0\ **Last updated:** [August 13, 2019, 4:11am UTC](https://discuss.elastic.co/t/logstash-got-response-code-403-contacting-elasticsearch-at-url-http-localhost-9200-logstash/194951 "2019-08-13T04:11:47Z")
Hi, I recently set security to elasticsearch. I configure elasticsearch by this doc And configure logstash by this doc Here is my logstash.yml: xpack.monitoring.enabled: true xpack.monitoring.elasticsearch.username… --- ## [Logstash crashed down the elasticsearch](https://discuss.elastic.co/t/logstash-crashed-down-the-elasticsearch/194935)
**Author:** [@Tianji\_Zhao](https://discuss.elastic.co/u/Tianji_Zhao)\ **Replies:** 4\ **Last updated:** [August 14, 2019, 12:40am UTC](https://discuss.elastic.co/t/logstash-crashed-down-the-elasticsearch/194935 "2019-08-14T00:40:02Z")
I use a logstash to fetching the data from the elasticsearch. The logstash works well when I use an empty filter. But it crashes the remote elastic when I add a json filter. The logstash.cong is as follow: input{ ela… --- ## [Remove part of message string in logstash config](https://discuss.elastic.co/t/remove-part-of-message-string-in-logstash-config/194913)
**Author:** [@rohitg](https://discuss.elastic.co/u/rohitg)\ **Replies:** 5\ **Last updated:** [August 13, 2019, 11:19pm UTC](https://discuss.elastic.co/t/remove-part-of-message-string-in-logstash-config/194913 "2019-08-13T23:19:36Z")
Hi, I have a string like - 123RGT78.XY.ABD.COM I need only 123RGT78 from this string and want to remove everything coming after the first dot (.) How to achieve this. filter { mutate { gsub =\> \["message", "\] } } --- ## [Logstash won't change template](https://discuss.elastic.co/t/logstash-wont-change-template/195098)
**Author:** [@jwalls](https://discuss.elastic.co/u/jwalls)\ **Replies:** 0\ **Last updated:** [August 13, 2019, 8:23pm UTC](https://discuss.elastic.co/t/logstash-wont-change-template/195098 "2019-08-13T20:23:38Z")
I am running logstash on a docker container, and I need to put in the template i am using. I have tried a PUT command to my elasticsearch URL, but the container continues to use the default template and gives me the fina… --- ## [Elastic training - Logstash data load](https://discuss.elastic.co/t/elastic-training-logstash-data-load/195097)
**Author:** [@Sangu](https://discuss.elastic.co/u/Sangu)\ **Replies:** 0\ **Last updated:** [August 13, 2019, 8:21pm UTC](https://discuss.elastic.co/t/elastic-training-logstash-data-load/195097 "2019-08-13T20:21:46Z")
I attended Elastic Engineer 1 trainig coursein Feb 2019, and was trying to load data into elastic from postgress using logstash, which is not loading and neither there is any error message. Training was on Elasticseacrh… --- ## [\[Index Lifecycle Management\] Dynamic rollover alias and template name](https://discuss.elastic.co/t/index-lifecycle-management-dynamic-rollover-alias-and-template-name/169614)
**Author:** [@Cyril\_Berber](https://discuss.elastic.co/u/Cyril_Berber)\ **Replies:** 7\ **Last updated:** [August 13, 2019, 7:28pm UTC](https://discuss.elastic.co/t/index-lifecycle-management-dynamic-rollover-alias-and-template-name/169614 "2019-08-13T19:28:17Z")
Use Case: ELK pipeline with Filebeat \> Logstash \> Elasticsearch Many Filebeat instances read different logs files and Docker output Logstash processes messages to parse and route them in different indices Use ILM on th… --- ## [Recommended logic for handling different log files on same host?](https://discuss.elastic.co/t/recommended-logic-for-handling-different-log-files-on-same-host/195040)
**Author:** [@hueyg](https://discuss.elastic.co/u/hueyg)\ **Replies:** 5\ **Last updated:** [August 13, 2019, 4:57pm UTC](https://discuss.elastic.co/t/recommended-logic-for-handling-different-log-files-on-same-host/195040 "2019-08-13T16:57:03Z")
I have a host with three different log files. Using one instance of filebeat to ship all of the logs. Started this process back on 6.2 and discovered the "type" option. So based on the log file path I would set the ty… --- ## [Logstash wildcards in input file path not processing files](https://discuss.elastic.co/t/logstash-wildcards-in-input-file-path-not-processing-files/194921)
**Author:** [@mtudisco](https://discuss.elastic.co/u/mtudisco)\ **Replies:** 6\ **Last updated:** [August 13, 2019, 3:32pm UTC](https://discuss.elastic.co/t/logstash-wildcards-in-input-file-path-not-processing-files/194921 "2019-08-13T15:32:44Z")
Hi, i have a logstash configuration like this: file { path =\> "/opt/audit\_logs\_modsec/xx/\*/\*/\*" # path =\> "/opt/audit\_logs\_modsec/xx/20171231/20171231-2358/\*" sincedb\_path =\> "/dev/null" … --- ## [Migrating logstash filters to ECS](https://discuss.elastic.co/t/migrating-logstash-filters-to-ecs/193866)
**Author:** [@Mich-b](https://discuss.elastic.co/u/Mich-b)\ **Replies:** 2\ **Last updated:** [August 13, 2019, 2:46pm UTC](https://discuss.elastic.co/t/migrating-logstash-filters-to-ecs/193866 "2019-08-13T14:46:19Z")
I want to migrate my custom Logstash pipeline to the Elastic Common Scheme, since I want to get started with the Elastic SIEM UI. In https://www.elastic.co/blog/migrating-to-elastic-common-schema-in-beats-environments i… --- ## [Extract string that starts with specific string using Grok](https://discuss.elastic.co/t/extract-string-that-starts-with-specific-string-using-grok/195048)
**Author:** [@Siri\_sour](https://discuss.elastic.co/u/Siri_sour)\ **Replies:** 0\ **Last updated:** [August 13, 2019, 2:41pm UTC](https://discuss.elastic.co/t/extract-string-that-starts-with-specific-string-using-grok/195048 "2019-08-13T14:41:30Z")
Hi everyone, I am trying to structure this log : --- ## [Traverse through XML document](https://discuss.elastic.co/t/traverse-through-xml-document/194967)
**Author:** [@mvenkat\_in](https://discuss.elastic.co/u/mvenkat_in)\ **Replies:** 1\ **Last updated:** [August 13, 2019, 2:30pm UTC](https://discuss.elastic.co/t/traverse-through-xml-document/194967 "2019-08-13T14:30:55Z")
Hi I have the below XML structure with a list of cartItem (it can be N and not definite in number). I want help for the below queries 1 - I want to traverse through each of the cartItem and check the conditions based … --- ## [Service Logstash with exit code 1 when restarting](https://discuss.elastic.co/t/service-logstash-with-exit-code-1-when-restarting/194944)
**Author:** [@shwesinhan](https://discuss.elastic.co/u/shwesinhan)\ **Replies:** 2\ **Last updated:** [August 13, 2019, 1:17pm UTC](https://discuss.elastic.co/t/service-logstash-with-exit-code-1-when-restarting/194944 "2019-08-13T13:17:56Z")
hi community I need advise from community about logstash I create logstash as a service using NSSM in Windows Server 2016. Then I usually use(start/stop/restart) it via services panel. Now I can not restart anymore. … --- ## [Logstash to ECS](https://discuss.elastic.co/t/logstash-to-ecs/193994)
**Author:** [@tgdesrochers](https://discuss.elastic.co/u/tgdesrochers)\ **Replies:** 4\ **Last updated:** [August 13, 2019, 1:16pm UTC](https://discuss.elastic.co/t/logstash-to-ecs/193994 "2019-08-13T13:16:50Z")
Since the beats modules can collect data and emit ECS format, is it possible to send data to a logstash node that contains the same data and have logstash perform the ECS conversion work. Example: legacy systems utilize… --- ## [Logstash/ElasticSearch/Kibana 7.3 not working for Netflow](https://discuss.elastic.co/t/logstash-elasticsearch-kibana-7-3-not-working-for-netflow/194829)
**Author:** [@Elena\_Kaidrikov](https://discuss.elastic.co/u/Elena_Kaidrikov)\ **Replies:** 2\ **Last updated:** [August 13, 2019, 1:05pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-kibana-7-3-not-working-for-netflow/194829 "2019-08-13T13:05:31Z")
Hi, I installed Logstash, ElasticSearch, Kibana, and ElasticFlow my logstash config : output { elasticsearch { hosts =\> \["localhost:9200"\] manage\_template =\> false index =\> "%{\[@metadata\]\[netflow\]}-%{\[@metadata\]\[v… --- ## [Create one doc per timestamp with this json](https://discuss.elastic.co/t/create-one-doc-per-timestamp-with-this-json/195000)
**Author:** [@igor.alvarez.lanit](https://discuss.elastic.co/u/igor.alvarez.lanit)\ **Replies:** 1\ **Last updated:** [August 13, 2019, 1:03pm UTC](https://discuss.elastic.co/t/create-one-doc-per-timestamp-with-this-json/195000 "2019-08-13T13:03:57Z")
Hi, I have this json and i need to create two docs, one per timestamp. It is possible? \[ { "timestamp": "23:55:48T30-JAN-2019", "filename": "xxxxx.nmon", "host": "xxxxx", "start\_date": "30-JAN-2019", "start\_time": "00:0… --- ## [Configuring Proxy in HTTP OUT Plugin](https://discuss.elastic.co/t/configuring-proxy-in-http-out-plugin/194785)
**Author:** [@rameshpasunoori](https://discuss.elastic.co/u/rameshpasunoori)\ **Replies:** 3\ **Last updated:** [August 13, 2019, 12:59pm UTC](https://discuss.elastic.co/t/configuring-proxy-in-http-out-plugin/194785 "2019-08-13T12:59:34Z")
Version: 7.3 Operating System: Windows 10 Config File (if you have sensitive info, please remove it): I tried three options to configure proxy for http out plugin. All these are not working, Please provide some input. … --- ## [Output based on metadata field is not working](https://discuss.elastic.co/t/output-based-on-metadata-field-is-not-working/194787)
**Author:** [@rameshpasunoori](https://discuss.elastic.co/u/rameshpasunoori)\ **Replies:** 3\ **Last updated:** [August 13, 2019, 12:50pm UTC](https://discuss.elastic.co/t/output-based-on-metadata-field-is-not-working/194787 "2019-08-13T12:50:47Z")
Version: 7.3 Operating System: Windows 10 Config File (if you have sensitive info, please remove it): filter {mutate { add\_field =\> { "\[@metadata\]\[ingestInto\]" =\> "${INGEST\_INTO:true}" } } } output { file { … --- ## [Logstash configuration not creating index in elasticseach](https://discuss.elastic.co/t/logstash-configuration-not-creating-index-in-elasticseach/193540)
**Author:** [@Abdul\_Gaffar\_Shaikh](https://discuss.elastic.co/u/Abdul_Gaffar_Shaikh)\ **Replies:** 9\ **Last updated:** [August 13, 2019, 12:41pm UTC](https://discuss.elastic.co/t/logstash-configuration-not-creating-index-in-elasticseach/193540 "2019-08-13T12:41:42Z")
i am trying to create index based on quarter and each document is unique in elasticsearch via logstash. the data is coming from multiple xml files. there seems to be no error in configuration file and pipeline is started… --- ## [CSV Parsing error - illegal quoting in line 1](https://discuss.elastic.co/t/csv-parsing-error-illegal-quoting-in-line-1/194952)
**Author:** [@jy94](https://discuss.elastic.co/u/jy94)\ **Replies:** 4\ **Last updated:** [August 13, 2019, 12:41pm UTC](https://discuss.elastic.co/t/csv-parsing-error-illegal-quoting-in-line-1/194952 "2019-08-13T12:41:02Z")
Hello, when I trying to parse a csv file, this error message shows up. input { file { path =\> "/finance/BTC-USD.csv" start\_position =\> "beginning" } } The filter part of this file is commented out to indicate th… --- ## [Parsing rsyslog to logstash](https://discuss.elastic.co/t/parsing-rsyslog-to-logstash/194939)
**Author:** [@gracia](https://discuss.elastic.co/u/gracia)\ **Replies:** 1\ **Last updated:** [August 13, 2019, 12:13pm UTC](https://discuss.elastic.co/t/parsing-rsyslog-to-logstash/194939 "2019-08-13T12:13:46Z")
Is it possible for logstash to ingest rsyslog? How do I set it up? Thank you. --- ## [Jdbc\_static left joins in local\_lookups](https://discuss.elastic.co/t/jdbc-static-left-joins-in-local-lookups/193953)
**Author:** [@ErikWarming](https://discuss.elastic.co/u/ErikWarming)\ **Replies:** 4\ **Last updated:** [August 13, 2019, 11:48am UTC](https://discuss.elastic.co/t/jdbc-static-left-joins-in-local-lookups/193953 "2019-08-13T11:48:59Z")
I have connected to a database via logstash filter jdbc\_static and input jdbc I have loaded the tables and defined the local\_db\_objects objects. I have created a query in the local\_lookups and when I just join tables t… --- ## [Logstash 6.4.2 java.lang.IllegalStateException (LoadError) Unsupported platform: x86\_64-windows](https://discuss.elastic.co/t/logstash-6-4-2-java-lang-illegalstateexception-loaderror-unsupported-platform-x86-64-windows/194991)
**Author:** [@vamsi11](https://discuss.elastic.co/u/vamsi11)\ **Replies:** 1\ **Last updated:** [August 13, 2019, 11:16am UTC](https://discuss.elastic.co/t/logstash-6-4-2-java-lang-illegalstateexception-loaderror-unsupported-platform-x86-64-windows/194991 "2019-08-13T11:16:38Z")
can anyone please suggest steps to overcome this unsupported platform error [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=492) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=494)