# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=523 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 524 --- ## [Logstash not error 400 while sending metrics to elasticsearch](https://discuss.elastic.co/t/logstash-not-error-400-while-sending-metrics-to-elasticsearch/186973)
**Author:** [@spavez](https://discuss.elastic.co/u/spavez)\ **Replies:** 0\ **Last updated:** [June 22, 2019, 8:35pm UTC](https://discuss.elastic.co/t/logstash-not-error-400-while-sending-metrics-to-elasticsearch/186973 "2019-06-22T20:35:14Z")
Monitoring information from logstash to elasticsearch is not working. Here is my configuration of logstash.yml and the error log thrown: Logstash.yml: path.config: /usr/share/logstash/conf.d xpack.monitoring.enabled: t… --- ## [Writing to sincedb\_ptah using Logstash and docker doesn't work](https://discuss.elastic.co/t/writing-to-sincedb-ptah-using-logstash-and-docker-doesnt-work/186605)
**Author:** [@EZprogramming](https://discuss.elastic.co/u/EZprogramming)\ **Replies:** 2\ **Last updated:** [June 22, 2019, 6:43pm UTC](https://discuss.elastic.co/t/writing-to-sincedb-ptah-using-logstash-and-docker-doesnt-work/186605 "2019-06-22T18:43:51Z")
Hi everyone, Background Information: I have a dockerfile that creates an image for me to run my Logstash. Using this method will not write a .sincedb\_\* file which is my problem. In my past experiences with Logstash, I … --- ## [Split path and take certain value by its position \[2\] and insert it into field](https://discuss.elastic.co/t/split-path-and-take-certain-value-by-its-position-2-and-insert-it-into-field/186639)
**Author:** [@nitzanm](https://discuss.elastic.co/u/nitzanm)\ **Replies:** 2\ **Last updated:** [June 22, 2019, 5:59pm UTC](https://discuss.elastic.co/t/split-path-and-take-certain-value-by-its-position-2-and-insert-it-into-field/186639 "2019-06-22T17:59:35Z")
hi... I need to extract from a path certain value by position... example: let's say I split this following path into array using '\\' as split char E:\\OUM82\\APP\\Logs\\UploadManager\_20062019.log I get: \[0\]=E: \[1\]=OU… --- ## [Logstash monitoring is not shown in kibana monitoring UI](https://discuss.elastic.co/t/logstash-monitoring-is-not-shown-in-kibana-monitoring-ui/180739)
**Author:** [@siddaram\_kj](https://discuss.elastic.co/u/siddaram_kj)\ **Replies:** 40\ **Last updated:** [June 22, 2019, 7:45am UTC](https://discuss.elastic.co/t/logstash-monitoring-is-not-shown-in-kibana-monitoring-ui/180739 "2019-06-22T07:45:29Z")
Hi i am using logstash version 7.0 and i set the x-pack monitoring for logstash , but the monitoring is not shown in the kibana monitoring UI and even the monitoring indices for logstash are also not getting created. be… --- ## [Unable to load csv file through logstash on windows](https://discuss.elastic.co/t/unable-to-load-csv-file-through-logstash-on-windows/186425)
**Author:** [@Muhammad\_Irfan](https://discuss.elastic.co/u/Muhammad_Irfan)\ **Replies:** 2\ **Last updated:** [June 22, 2019, 6:11am UTC](https://discuss.elastic.co/t/unable-to-load-csv-file-through-logstash-on-windows/186425 "2019-06-22T06:11:00Z")
Hello Everyone, I am new in ELK, i was trying to load simple CSV file on logstash but got error although, i change sincedb\_path option and also local host but same result received.. so please guide me my input file i… --- ## [Is it possible to write collected data (using logstash in a database) becuase all the collected data is removed after stopping logstash](https://discuss.elastic.co/t/is-it-possible-to-write-collected-data-using-logstash-in-a-database-becuase-all-the-collected-data-is-removed-after-stopping-logstash/186943)
**Author:** [@kumar4](https://discuss.elastic.co/u/kumar4)\ **Replies:** 0\ **Last updated:** [June 22, 2019, 1:25am UTC](https://discuss.elastic.co/t/is-it-possible-to-write-collected-data-using-logstash-in-a-database-becuase-all-the-collected-data-is-removed-after-stopping-logstash/186943 "2019-06-22T01:25:49Z")
i am using logstash to collect data from syslogs and social network (using "logstash -f" command) but i see that when i shutdown logstash all the data is deleted, how i can persist data so that i can it can be retrieved … --- ## [Suggestions for S3 input shortcomings, buffering & durability, & redis](https://discuss.elastic.co/t/suggestions-for-s3-input-shortcomings-buffering-durability-redis/186942)
**Author:** [@abcarroll](https://discuss.elastic.co/u/abcarroll)\ **Replies:** 0\ **Last updated:** [June 22, 2019, 1:03am UTC](https://discuss.elastic.co/t/suggestions-for-s3-input-shortcomings-buffering-durability-redis/186942 "2019-06-22T01:03:42Z")
Several small issues here. With some common themes. We've been using ElasticSearch for about 18 months for general website search, however this week I setup our first logstash instance and began processing processing a… --- ## [Collecting details about visited website and user connection informations from my router](https://discuss.elastic.co/t/collecting-details-about-visited-website-and-user-connection-informations-from-my-router/186937)
**Author:** [@kumar4](https://discuss.elastic.co/u/kumar4)\ **Replies:** 0\ **Last updated:** [June 21, 2019, 11:06pm UTC](https://discuss.elastic.co/t/collecting-details-about-visited-website-and-user-connection-informations-from-my-router/186937 "2019-06-21T23:06:10Z")
hi dears, i have a router connecting usres to external network and i want to know if it is possible collect details about visited website and user coonection information from my router ? are these info avalaible on my ro… --- ## [Regarding illegal\_argument\_exception and empty value](https://discuss.elastic.co/t/regarding-illegal-argument-exception-and-empty-value/186922)
**Author:** [@leeyu](https://discuss.elastic.co/u/leeyu)\ **Replies:** 5\ **Last updated:** [June 21, 2019, 8:01pm UTC](https://discuss.elastic.co/t/regarding-illegal-argument-exception-and-empty-value/186922 "2019-06-21T20:01:24Z")
Hi all, I got many warning messages like this \[2019-06-21T10:34:01,663\]\[WARN \]\[logstash.outputs.elasticsearch\] Could not index event to Elasticsearch. {:status=\>400, :action=\>\["index", {:\_id=\>nil, :\_index=\>"cv\_threat\_… --- ## [Avro files as input](https://discuss.elastic.co/t/avro-files-as-input/186929)
**Author:** [@mj2019](https://discuss.elastic.co/u/mj2019)\ **Replies:** 0\ **Last updated:** [June 21, 2019, 7:28pm UTC](https://discuss.elastic.co/t/avro-files-as-input/186929 "2019-06-21T19:28:10Z")
Hi, I am a newbie to Logstash. Is there a way I can use avro files (not avro record directly from Kafka) as input to Logstash? --- ## [Pipeline output isolator pattern not working](https://discuss.elastic.co/t/pipeline-output-isolator-pattern-not-working/186910)
**Author:** [@erikmathis](https://discuss.elastic.co/u/erikmathis)\ **Replies:** 0\ **Last updated:** [June 21, 2019, 4:00pm UTC](https://discuss.elastic.co/t/pipeline-output-isolator-pattern-not-working/186910 "2019-06-21T16:00:49Z")
I just converted a single pipeline to the mutli-pipeline as suggested here When I stopped elasticsearch the file output stops working, the same problem as when it was a single pipeline. Am I missing something here? Log… --- ## [Logstash failed to process cluster event (put-mapping) within 30s](https://discuss.elastic.co/t/logstash-failed-to-process-cluster-event-put-mapping-within-30s/186129)
**Author:** [@Rajeev\_Prakash](https://discuss.elastic.co/u/Rajeev_Prakash)\ **Replies:** 4\ **Last updated:** [June 21, 2019, 1:58pm UTC](https://discuss.elastic.co/t/logstash-failed-to-process-cluster-event-put-mapping-within-30s/186129 "2019-06-21T13:58:04Z")
logstash failed to process cluster event (put-mapping) within 30s Thanks for the help in advance --- ## [Parsing apache airflow logs](https://discuss.elastic.co/t/parsing-apache-airflow-logs/186792)
**Author:** [@e526e9a7b99f61623fe7](https://discuss.elastic.co/u/e526e9a7b99f61623fe7)\ **Replies:** 2\ **Last updated:** [June 21, 2019, 1:55pm UTC](https://discuss.elastic.co/t/parsing-apache-airflow-logs/186792 "2019-06-21T13:55:21Z")
My main goal is to parse apache airflow logs into particular fields using logstash, feed it into elasticsearch and visualise them using kibana. There is no particular grok pattern available for airflow logs. I'm fairly n… --- ## [Consommation de CPU élevée](https://discuss.elastic.co/t/consommation-de-cpu-elevee/186890)
**Author:** [@nono](https://discuss.elastic.co/u/nono)\ **Replies:** 0\ **Last updated:** [June 21, 2019, 1:51pm UTC](https://discuss.elastic.co/t/consommation-de-cpu-elevee/186890 "2019-06-21T13:51:27Z")
La consommation de CPU de logstash installé sous LiNUX est élevée. Est-il possible de la réduire? --- ## [Beats input plugin cannot read password protected ssl key](https://discuss.elastic.co/t/beats-input-plugin-cannot-read-password-protected-ssl-key/186771)
**Author:** [@tterranigma](https://discuss.elastic.co/u/tterranigma)\ **Replies:** 2\ **Last updated:** [June 21, 2019, 1:50pm UTC](https://discuss.elastic.co/t/beats-input-plugin-cannot-read-password-protected-ssl-key/186771 "2019-06-21T13:50:00Z")
I am trying to setup the beats input to use ssl to authenticate to the connecting beats client. This is what I have in my beats.conf file: # vi:ft=logstash.jinja2 input { beats { id =\> 'beat-input' port =\> 504… --- ## [Redis input pipeline intermittently stops for couple minutes](https://discuss.elastic.co/t/redis-input-pipeline-intermittently-stops-for-couple-minutes/185400)
**Author:** [@YvorL](https://discuss.elastic.co/u/YvorL)\ **Replies:** 6\ **Last updated:** [June 21, 2019, 1:43pm UTC](https://discuss.elastic.co/t/redis-input-pipeline-intermittently-stops-for-couple-minutes/185400 "2019-06-21T13:43:41Z")
Hi! One of my "heavy duty" pipelines started acting strangely in the past weeks. I get alerts that ingesting stopped then I see that after a spike everything goes back to normal. I checked Elastic and Logstash logs on a… --- ## [Logstash creating multi GB syslog log file & saving queries output in plain text!](https://discuss.elastic.co/t/logstash-creating-multi-gb-syslog-log-file-saving-queries-output-in-plain-text/186784)
**Author:** [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Replies:** 2\ **Last updated:** [June 21, 2019, 1:40pm UTC](https://discuss.elastic.co/t/logstash-creating-multi-gb-syslog-log-file-saving-queries-output-in-plain-text/186784 "2019-06-21T13:40:47Z")
Hi, Yesterday I started using a config that utilizes the elasticsearch input to run a query. However I noticed the following rather insane behavior: All the output received from elastic after Logstash runs the query i… --- ## [Logstash input file not working](https://discuss.elastic.co/t/logstash-input-file-not-working/186713)
**Author:** [@paksoft18](https://discuss.elastic.co/u/paksoft18)\ **Replies:** 5\ **Last updated:** [June 21, 2019, 1:36pm UTC](https://discuss.elastic.co/t/logstash-input-file-not-working/186713 "2019-06-21T13:36:39Z")
Hello, Greetings, I'm new to ELK Stack and I'm trying to put some router logs to elasticsearch via Logstash but it's not working. it doesn't create the index on elasticsearch please help. My Config file is as below i… --- ## [Sequential record pull from JDBC to Elasticsearch using Logstash is out of the sequence](https://discuss.elastic.co/t/sequential-record-pull-from-jdbc-to-elasticsearch-using-logstash-is-out-of-the-sequence/186864)
**Author:** [@saif3r](https://discuss.elastic.co/u/saif3r)\ **Replies:** 4\ **Last updated:** [June 21, 2019, 1:32pm UTC](https://discuss.elastic.co/t/sequential-record-pull-from-jdbc-to-elasticsearch-using-logstash-is-out-of-the-sequence/186864 "2019-06-21T13:32:59Z")
Hello All, I am facing an issue while pulling data from SQL Database. It appears that the sequential document population which is based on an ID is sometimes out of the sequence... Let me explain it on an actual example… --- ## [Jdbc streaming and sql error](https://discuss.elastic.co/t/jdbc-streaming-and-sql-error/186766)
**Author:** [@jonathan01](https://discuss.elastic.co/u/jonathan01)\ **Replies:** 1\ **Last updated:** [June 21, 2019, 1:08pm UTC](https://discuss.elastic.co/t/jdbc-streaming-and-sql-error/186766 "2019-06-21T13:08:13Z")
I am using the jdbc streaming and below is my code jdbc\_streaming { jdbc\_driver\_library =\> "C:\\logstash-7.0.0\\bin\\sqljdbc42.jar" jdbc\_driver\_class =\> "com.microsoft.sqlserver.jdbc.SQLServerDriver" jdbc\_connection\_str… --- ## [Separate country code from phone number](https://discuss.elastic.co/t/separate-country-code-from-phone-number/186075)
**Author:** [@Andreasky](https://discuss.elastic.co/u/Andreasky)\ **Replies:** 2\ **Last updated:** [June 21, 2019, 12:56pm UTC](https://discuss.elastic.co/t/separate-country-code-from-phone-number/186075 "2019-06-21T12:56:23Z")
How can I separate the country code from a phone number? I want +4712345678 to be separated into two terms: "country code" =\> "+47", "phone number" =\> "12345678" --- ## [Logstash count API rather than \_search API](https://discuss.elastic.co/t/logstash-count-api-rather-than-search-api/186859)
**Author:** [@pastechecker](https://discuss.elastic.co/u/pastechecker)\ **Replies:** 0\ **Last updated:** [June 21, 2019, 10:00am UTC](https://discuss.elastic.co/t/logstash-count-api-rather-than-search-api/186859 "2019-06-21T10:00:50Z")
Hello. Is there a way to use \_count API in logstash input elasticsearch plugin rather than the default \_search one? I know that I can solve this by using exec plugin and some custom curl command, but I wonder if there … --- ## [Ruby filter not running all the code on every event?!](https://discuss.elastic.co/t/ruby-filter-not-running-all-the-code-on-every-event/186846)
**Author:** [@Miguel\_Oliveira](https://discuss.elastic.co/u/Miguel_Oliveira)\ **Replies:** 0\ **Last updated:** [June 21, 2019, 9:23am UTC](https://discuss.elastic.co/t/ruby-filter-not-running-all-the-code-on-every-event/186846 "2019-06-21T09:23:18Z")
Hello everyone, I'm implementing a use case where I'm reading events from a Kafka topic and then I use a Ruby filter to run some custom code (version 6.6.2). The idea is to find some date fields that can have multiple … --- ## [Logstash jdbc call store procedure oracle](https://discuss.elastic.co/t/logstash-jdbc-call-store-procedure-oracle/186833)
**Author:** [@RPa](https://discuss.elastic.co/u/RPa)\ **Replies:** 0\ **Last updated:** [June 21, 2019, 8:38am UTC](https://discuss.elastic.co/t/logstash-jdbc-call-store-procedure-oracle/186833 "2019-06-21T08:38:36Z")
Hello, I created some procedure in oracle database aum which return sys\_refcursor. create or replace procedure aum IS v\_code NUMBER; v\_errm VARCHAR2(200); rc sys\_refcursor; BEGIN open rc for select username fro… --- ## [Logstash - How to split this field](https://discuss.elastic.co/t/logstash-how-to-split-this-field/186468)
**Author:** [@Raj\_Kumar](https://discuss.elastic.co/u/Raj_Kumar)\ **Replies:** 2\ **Last updated:** [June 21, 2019, 8:18am UTC](https://discuss.elastic.co/t/logstash-how-to-split-this-field/186468 "2019-06-21T08:18:43Z")
Hi There, Could anyone help me to split the field in two different fields and values event\_data.Hashes : MD5=F9DFBDD2BA6AE341FE1FF7585B4,SHA256=36D007B5FE48FE7C27FB01085E2E322BA9EC78EAABC6DDE506C308D1 is it possible… --- ## [Configuring logstash with ssl and sending message to kafka(with ssl) topic](https://discuss.elastic.co/t/configuring-logstash-with-ssl-and-sending-message-to-kafka-with-ssl-topic/186805)
**Author:** [@rchillal](https://discuss.elastic.co/u/rchillal)\ **Replies:** 0\ **Last updated:** [June 21, 2019, 6:51am UTC](https://discuss.elastic.co/t/configuring-logstash-with-ssl-and-sending-message-to-kafka-with-ssl-topic/186805 "2019-06-21T06:51:24Z")
I have logstash-6.5.4 (with ssl) in my local (hostname: webbox) and kafka-2.0 (with ssl) on another (hostname: kafkabox). I am not able to receive message in kafka topic when message is sent from logstash. Neither erro… --- ## [Why there's no agent field available once used %{COMBINEDAPACHELOG} in ELK Stack 7.1](https://discuss.elastic.co/t/why-theres-no-agent-field-available-once-used-combinedapachelog-in-elk-stack-7-1/186402)
**Author:** [@mark.quilates](https://discuss.elastic.co/u/mark.quilates)\ **Replies:** 2\ **Last updated:** [June 21, 2019, 5:36am UTC](https://discuss.elastic.co/t/why-theres-no-agent-field-available-once-used-combinedapachelog-in-elk-stack-7-1/186402 "2019-06-21T05:36:12Z")
Hi Any idea why there's no "agent" field available once used %{COMBINEDAPACHELOG} in ELK Stack 7.1? Since what to use useragent: useragent { source =\> "agent" } Any idea how? --- ## [Having issues parsing json](https://discuss.elastic.co/t/having-issues-parsing-json/183619)
**Author:** [@az123](https://discuss.elastic.co/u/az123)\ **Replies:** 14\ **Last updated:** [June 21, 2019, 1:39am UTC](https://discuss.elastic.co/t/having-issues-parsing-json/183619 "2019-06-21T01:39:22Z")
I am seeing \_jsonparsefailure when using json codec in logstash. "message" =\> "%{\\"node\_id\_str\\":\\"xxxxxxx\\",\\"subscription\_id\_str\\":\\"Sub1\\",\\"encoding\_path\\":\\"Cisco-IOS-XR-fib-common-oper:fib/nodes/node/protocols/… --- ## [Unstructured data source](https://discuss.elastic.co/t/unstructured-data-source/186316)
**Author:** [@ashokchinna](https://discuss.elastic.co/u/ashokchinna)\ **Replies:** 12\ **Last updated:** [June 20, 2019, 9:47pm UTC](https://discuss.elastic.co/t/unstructured-data-source/186316 "2019-06-20T21:47:21Z")
I am trying to read data from an unstructured data source. What is the best filter to start with? Grok? Example:- The below log details should fall into one document. LogFile- 2019-06-17T00:00:01 ProcessId\[1234\] Tra… --- ## [Filtering nested json outupt](https://discuss.elastic.co/t/filtering-nested-json-outupt/186765)
**Author:** [@EZprogramming](https://discuss.elastic.co/u/EZprogramming)\ **Replies:** 4\ **Last updated:** [June 20, 2019, 9:44pm UTC](https://discuss.elastic.co/t/filtering-nested-json-outupt/186765 "2019-06-20T21:44:53Z")
JSON output: {"d":\[{"id":960929,"created\_at":"2019-06-19T00:27:28.107Z","belongs\_to\_\_application":{"\_\_deferred":{"uri":"/resin/application(1415733)"},"\_\_id":1415733},"is\_created\_by\_\_user":{"\_\_deferred":{"uri":"/resin/us… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=522) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=524)