# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=529 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 530 --- ## [Logstash cipher filter issue](https://discuss.elastic.co/t/logstash-cipher-filter-issue/185549)
**Author:** [@yogender20](https://discuss.elastic.co/u/yogender20)\ **Replies:** 0\ **Last updated:** [June 13, 2019, 6:36am UTC](https://discuss.elastic.co/t/logstash-cipher-filter-issue/185549 "2019-06-13T06:36:06Z")
Hello, I'm encrypting some data using logstash cipher filter & store the encrypted data in elasticsearch. The code snippet used for the encryption is as follow: filter { cipher { algorithm =\> "aes-256-cbc" cipher\_pa… --- ## [Input from kafka to logstash](https://discuss.elastic.co/t/input-from-kafka-to-logstash/184738)
**Author:** [@Harsha606](https://discuss.elastic.co/u/Harsha606)\ **Replies:** 4\ **Last updated:** [June 13, 2019, 9:20am UTC](https://discuss.elastic.co/t/input-from-kafka-to-logstash/184738 "2019-06-13T09:20:07Z")
Hi All, my architecture is as below filebeat -\> kafka -\> logstash -\> elastic from, filebeat i am successfully pushing logs to kafka, what i observed in kafka topic message is log":{"file":{"path":"/mnt/volume0/XXXX-v2… --- ## [After upgrading logstash 6.7.0 to 7.0.1 continously restarting](https://discuss.elastic.co/t/after-upgrading-logstash-6-7-0-to-7-0-1-continously-restarting/185161)
**Author:** [@sharath\_1245](https://discuss.elastic.co/u/sharath_1245)\ **Replies:** 2\ **Last updated:** [June 13, 2019, 8:59am UTC](https://discuss.elastic.co/t/after-upgrading-logstash-6-7-0-to-7-0-1-continously-restarting/185161 "2019-06-13T08:59:10Z")
Hi Team, after upgrading logstash 6.7.0 to 7.0.1 unable to start logstash. we are getting below error. Hi Team, after upgrading logstash 6.7.0 to 7.0.1 unable to start logstash. we are getting below error. \[2019-06… --- ## [Jsonparsefailure becoz of \\/](https://discuss.elastic.co/t/jsonparsefailure-becoz-of/185553)
**Author:** [@bhupesh\_awasthi](https://discuss.elastic.co/u/bhupesh_awasthi)\ **Replies:** 0\ **Last updated:** [June 13, 2019, 6:47am UTC](https://discuss.elastic.co/t/jsonparsefailure-becoz-of/185553 "2019-06-13T06:47:15Z")
\<{"index":{"\_id":1}} {"name":"Wine - Maipo Valle Cabernet","price":152,"in\_stock":38,"sold":47,"tags":\["Alcohol","Wine"\],"description":"Aliquam augue quam, sollicitudin vitae, consectetuer eget, rutrum at, lorem. Intege… --- ## [Logstash issue after implementing xpack](https://discuss.elastic.co/t/logstash-issue-after-implementing-xpack/185184)
**Author:** [@Tek\_Chand](https://discuss.elastic.co/u/Tek_Chand)\ **Replies:** 2\ **Last updated:** [June 13, 2019, 3:38am UTC](https://discuss.elastic.co/t/logstash-issue-after-implementing-xpack/185184 "2019-06-13T03:38:21Z")
Hello Team, We are using ELK6.4.4 and everything is working fine. But now we are trying ELK7.1 for RBAC and TLS feature. We have implement xpack security in elasticserach. We have added the below 2 line in elasticsear… --- ## [Regex kind of expression for path in logstash 2.4](https://discuss.elastic.co/t/regex-kind-of-expression-for-path-in-logstash-2-4/185263)
**Author:** [@Ram\_Sharan\_Mittal](https://discuss.elastic.co/u/Ram_Sharan_Mittal)\ **Replies:** 1\ **Last updated:** [June 13, 2019, 1:23am UTC](https://discuss.elastic.co/t/regex-kind-of-expression-for-path-in-logstash-2-4/185263 "2019-06-13T01:23:22Z")
logstash 2.4 getting used, Can pattern like path =\> \["/var/log/console/\*console/sta/\*console-\*"\]. Can be define in file input .. I doubt multiple \* in one segment (e.g. \*console-\*) not supported. There are multiple file … --- ## [Logstash를 통한 oracle jdbc input 한글깨짐](https://discuss.elastic.co/t/logstash-oracle-jdbc-input/185326)
**Author:** [@adorable\_man](https://discuss.elastic.co/u/adorable_man)\ **Replies:** 1\ **Last updated:** [June 13, 2019, 1:08am UTC](https://discuss.elastic.co/t/logstash-oracle-jdbc-input/185326 "2019-06-13T01:08:36Z")
elasticsearch version : 6.5.4 database : oracle MSG : 000154190612홍길동 input { jdbc { jdbc\_connect\_string =\> "jdbc:oracle:thin:@xx.xx.xxx.xx:xxxxx/TEST" jdbc\_driver\_class =\> "Java::oracle:jdbc.driver.Oracle… --- ## [Logstash \_grokparsefailure in same pattern of data](https://discuss.elastic.co/t/logstash-grokparsefailure-in-same-pattern-of-data/185423)
**Author:** [@ranjana.ghimire](https://discuss.elastic.co/u/ranjana.ghimire)\ **Replies:** 1\ **Last updated:** [June 13, 2019, 12:06am UTC](https://discuss.elastic.co/t/logstash-grokparsefailure-in-same-pattern-of-data/185423 "2019-06-13T00:06:34Z")
Hi , The strange thing is happenning in logstash. For the same format of log, it gets parsed quite easily however sometimes its throwing grokparsefailure . Pattern: grok { match =\> { "message" =\> '%{WORD:logtit… --- ## [How to make CSV headers and Keys the column names in JSON output?](https://discuss.elastic.co/t/how-to-make-csv-headers-and-keys-the-column-names-in-json-output/185513)
**Author:** [@EZprogramming](https://discuss.elastic.co/u/EZprogramming)\ **Replies:** 1\ **Last updated:** [June 12, 2019, 11:15pm UTC](https://discuss.elastic.co/t/how-to-make-csv-headers-and-keys-the-column-names-in-json-output/185513 "2019-06-12T23:15:07Z")
Given this CSV file which is a mix of column values + key, value pairs. How would you filter these information such that the headers at the top of the CSV become the column names, and the Keys in key, value pairs become … --- ## [Elastic search index is getting created as BSON instead of JSON](https://discuss.elastic.co/t/elastic-search-index-is-getting-created-as-bson-instead-of-json/185222)
**Author:** [@nagarajanj](https://discuss.elastic.co/u/nagarajanj)\ **Replies:** 2\ **Last updated:** [June 12, 2019, 9:54pm UTC](https://discuss.elastic.co/t/elastic-search-index-is-getting-created-as-bson-instead-of-json/185222 "2019-06-12T21:54:22Z")
We are using logstash to move the data b/w mongo db and elastic.. however the data is getting created as BSON instead of JSON.. I keep getting all the data in a "log\_entry" field as one big string. when we do the same… --- ## [Logstash consumers from Kafka never increase beyond a certain count](https://discuss.elastic.co/t/logstash-consumers-from-kafka-never-increase-beyond-a-certain-count/185519)
**Author:** [@chakragod](https://discuss.elastic.co/u/chakragod)\ **Replies:** 0\ **Last updated:** [June 12, 2019, 9:45pm UTC](https://discuss.elastic.co/t/logstash-consumers-from-kafka-never-increase-beyond-a-certain-count/185519 "2019-06-12T21:45:28Z")
Here is our config: 64 replicas of Logstash Deployments (run ELK in Kubernetes cluster) 64 partitions on the Kafka topic 1 consumer thread per Logstash Resources for Logstash: Heap size: 3g Cpu: 1000m (per logstash … --- ## [CSV files have different columns](https://discuss.elastic.co/t/csv-files-have-different-columns/185063)
**Author:** [@EZprogramming](https://discuss.elastic.co/u/EZprogramming)\ **Replies:** 15\ **Last updated:** [June 12, 2019, 7:08pm UTC](https://discuss.elastic.co/t/csv-files-have-different-columns/185063 "2019-06-12T19:08:54Z")
Hi everyone, Issue: I have two kinds CSV file, one kind (default) has two columns, and the other one (custom) has about seventeen columns. Default CSV: rosbagTimestamp data 1.56E+18 53829632 1.56E+18 538296… --- ## [Elasticsearch Unreachable \[Manticore::SocketException\] Connection refused](https://discuss.elastic.co/t/elasticsearch-unreachable-manticore-socketexception-connection-refused/185462)
**Author:** [@matejt](https://discuss.elastic.co/u/matejt)\ **Replies:** 1\ **Last updated:** [June 12, 2019, 7:02pm UTC](https://discuss.elastic.co/t/elasticsearch-unreachable-manticore-socketexception-connection-refused/185462 "2019-06-12T19:02:56Z")
Hi, I have elasticsearch and logstash running on separate servers. Elasticsearch is up and running, and I'm able to issue a curl command from the server where logstash is installed: curl http://my\_ip\_address:9200 and I … --- ## [Data not going from Logstash to Elasticsearch (7.1.1)](https://discuss.elastic.co/t/data-not-going-from-logstash-to-elasticsearch-7-1-1/185495)
**Author:** [@topher](https://discuss.elastic.co/u/topher)\ **Replies:** 3\ **Last updated:** [June 12, 2019, 6:54pm UTC](https://discuss.elastic.co/t/data-not-going-from-logstash-to-elasticsearch-7-1-1/185495 "2019-06-12T18:54:34Z")
Since updating to 7.1.1 I have been unable to get data to Elasticsearch consistently. I posted about 429 errors the other day which seem to have gone away, but now I have no data coming into ES through Logstash. (Filebe… --- ## [Parse out window event logs message](https://discuss.elastic.co/t/parse-out-window-event-logs-message/185296)
**Author:** [@NieruHawic](https://discuss.elastic.co/u/NieruHawic)\ **Replies:** 4\ **Last updated:** [June 12, 2019, 5:38pm UTC](https://discuss.elastic.co/t/parse-out-window-event-logs-message/185296 "2019-06-12T17:38:18Z")
I would like to take the message section from windows event logs and be able to use the different areas to filter. Right now the message comes across as "message": "An attempt was made to access an object.\\n\\nSubject:\\n… --- ## [Logstash Monitoring APIs](https://discuss.elastic.co/t/logstash-monitoring-apis/185268)
**Author:** [@Jayanthi](https://discuss.elastic.co/u/Jayanthi)\ **Replies:** 1\ **Last updated:** [June 12, 2019, 5:11pm UTC](https://discuss.elastic.co/t/logstash-monitoring-apis/185268 "2019-06-12T17:11:00Z")
I am trying to use the monitoring APIs to determine if the logstash server is performing well and processing the input. What are the data that you would track to identify this from the API outputs that we get from Monito… --- ## [Logstash output changing field location randomly](https://discuss.elastic.co/t/logstash-output-changing-field-location-randomly/185448)
**Author:** [@mdavis00](https://discuss.elastic.co/u/mdavis00)\ **Replies:** 4\ **Last updated:** [June 12, 2019, 4:24pm UTC](https://discuss.elastic.co/t/logstash-output-changing-field-location-randomly/185448 "2019-06-12T16:24:44Z")
I am pulling in logs from Kafka and sending them out to Elasticsearch. I have been getting this set up over the last few weeks and everything seems to be working as expected. Today I noticed every time I start the serv… --- ## [Unable to pull from kineses using the kinesis logstash plugin](https://discuss.elastic.co/t/unable-to-pull-from-kineses-using-the-kinesis-logstash-plugin/185466)
**Author:** [@asrinivasan](https://discuss.elastic.co/u/asrinivasan)\ **Replies:** 0\ **Last updated:** [June 12, 2019, 4:11pm UTC](https://discuss.elastic.co/t/unable-to-pull-from-kineses-using-the-kinesis-logstash-plugin/185466 "2019-06-12T16:11:26Z")
Hi I have installed logstash 7.1.1 on centos 7, I have installed the kineseis plugin and codec plugin. the aws credentials and the perm to access the stream are all good what i see in the logs is and shows its connected… --- ## [Is it possible to post JSON to logstash directly outside of a BEAT?](https://discuss.elastic.co/t/is-it-possible-to-post-json-to-logstash-directly-outside-of-a-beat/185436)
**Author:** [@raged](https://discuss.elastic.co/u/raged)\ **Replies:** 2\ **Last updated:** [June 12, 2019, 3:39pm UTC](https://discuss.elastic.co/t/is-it-possible-to-post-json-to-logstash-directly-outside-of-a-beat/185436 "2019-06-12T15:39:41Z")
Good morning, Does anyone know if it's possible to post data to logstash via something like this: invoke-webrequest -uri $uri -ContentType 'application/json' -body $body We have our pipeline on logstash listening on p… --- ## [How to print events and fields in Logstash using Ruby code?](https://discuss.elastic.co/t/how-to-print-events-and-fields-in-logstash-using-ruby-code/185305)
**Author:** [@EZprogramming](https://discuss.elastic.co/u/EZprogramming)\ **Replies:** 2\ **Last updated:** [June 12, 2019, 2:34pm UTC](https://discuss.elastic.co/t/how-to-print-events-and-fields-in-logstash-using-ruby-code/185305 "2019-06-12T14:34:33Z")
Is there anyway I can print the events with ruby in Logstash? Logstash.conf file: input { http { port =\> 9600 id =\> "my\_plugin\_id" } } filter { ruby { code =\> "puts event\['message'\]… --- ## [Duplicating Events using JDBC](https://discuss.elastic.co/t/duplicating-events-using-jdbc/185431)
**Author:** [@BrunoSdeMenezes](https://discuss.elastic.co/u/BrunoSdeMenezes)\ **Replies:** 1\ **Last updated:** [June 12, 2019, 2:46pm UTC](https://discuss.elastic.co/t/duplicating-events-using-jdbc/185431 "2019-06-12T14:46:38Z")
Hello there, I'm using jdbc input plugin to create an index, but if I run the logstash config, close and open again, it duplicates the events on index. This is my script: input{ jdbc{ jdbc\_connection\_string =\> … --- ## [Logstash does not transfer all CSV files to an Elasticsearch index](https://discuss.elastic.co/t/logstash-does-not-transfer-all-csv-files-to-an-elasticsearch-index/185297)
**Author:** [@EZprogramming](https://discuss.elastic.co/u/EZprogramming)\ **Replies:** 1\ **Last updated:** [June 12, 2019, 2:37pm UTC](https://discuss.elastic.co/t/logstash-does-not-transfer-all-csv-files-to-an-elasticsearch-index/185297 "2019-06-12T14:37:36Z")
Hi everyone, Background info: I am trying to read CSV files' data and transfer it to an Elasticsearch index. Issue: Elasticsearch index shows only one CSV path in that specific index. I called it "rosindex". Refreshi… --- ## [Logstash Filters - Grok a email advice](https://discuss.elastic.co/t/logstash-filters-grok-a-email-advice/185349)
**Author:** [@gerard.ramos](https://discuss.elastic.co/u/gerard.ramos)\ **Replies:** 3\ **Last updated:** [June 12, 2019, 1:49pm UTC](https://discuss.elastic.co/t/logstash-filters-grok-a-email-advice/185349 "2019-06-12T13:49:49Z")
Hi, I was trying to grok a email log message to take the email address and make a field of it. I google it and i don't find the patterns to do it. Any help please?? Message examples: \*\*\*\* ALERT \*\*\*\* Failed IMAP authent… --- ## [Logstash Filter Conditional Usage](https://discuss.elastic.co/t/logstash-filter-conditional-usage/185284)
**Author:** [@Tony\_Chirillo](https://discuss.elastic.co/u/Tony_Chirillo)\ **Replies:** 2\ **Last updated:** [June 12, 2019, 1:36pm UTC](https://discuss.elastic.co/t/logstash-filter-conditional-usage/185284 "2019-06-12T13:36:58Z")
I am trying to code my filter to drop an event when three specific fields contain the text "Other", but cant seem to get events actually to drop. Here is the statement I am using, do I have the syntax and format correct?… --- ## [Regarding the XML file issue](https://discuss.elastic.co/t/regarding-the-xml-file-issue/185392)
**Author:** [@dilipchiru](https://discuss.elastic.co/u/dilipchiru)\ **Replies:** 2\ **Last updated:** [June 12, 2019, 1:28pm UTC](https://discuss.elastic.co/t/regarding-the-xml-file-issue/185392 "2019-06-12T13:28:49Z")
Hi , I have added the XML file configuration on the in the logstash where i sent the data to logstash and i can see the data is sent in the console and index is created in kibana but the data is not loaded in kibana and… --- ## [Logstash shutting down](https://discuss.elastic.co/t/logstash-shutting-down/184797)
**Author:** [@BrunoSdeMenezes](https://discuss.elastic.co/u/BrunoSdeMenezes)\ **Replies:** 2\ **Last updated:** [June 12, 2019, 1:21pm UTC](https://discuss.elastic.co/t/logstash-shutting-down/184797 "2019-06-12T13:21:20Z")
Hello, When I use a config file to create metrics on Logstash, after all calculations, it shuts down. But I need it keep running. This is my script, can someone say what I'm doing wrong? I 'm new to Logstash and first t… --- ## [Issue with json arrays](https://discuss.elastic.co/t/issue-with-json-arrays/185374)
**Author:** [@moti.mor](https://discuss.elastic.co/u/moti.mor)\ **Replies:** 1\ **Last updated:** [June 12, 2019, 1:19pm UTC](https://discuss.elastic.co/t/issue-with-json-arrays/185374 "2019-06-12T13:19:35Z")
Hi guys, i have issue with parssing json with logstash. here is my logstash configuration: input { beats { port =\> "5044" } } filter { ruby { init =\> " def arrays\_to\_hash(h) h.each do |k,v| # If v is nil, an… --- ## [Jdbc\_static and combining tables](https://discuss.elastic.co/t/jdbc-static-and-combining-tables/185155)
**Author:** [@ErikWarming](https://discuss.elastic.co/u/ErikWarming)\ **Replies:** 1\ **Last updated:** [June 12, 2019, 12:18pm UTC](https://discuss.elastic.co/t/jdbc-static-and-combining-tables/185155 "2019-06-12T12:18:03Z")
I am currently building experience with JDBC in logstash and I have this testing setup. I'm connecting to my own server where I get the backbone table from a SQL server ( \[dbo\].\[Numbers\] ) and the table that I want to jo… --- ## [Best practice running logstash on kubernetes](https://discuss.elastic.co/t/best-practice-running-logstash-on-kubernetes/184107)
**Author:** [@asp](https://discuss.elastic.co/u/asp)\ **Replies:** 1\ **Last updated:** [June 12, 2019, 12:09pm UTC](https://discuss.elastic.co/t/best-practice-running-logstash-on-kubernetes/184107 "2019-06-12T12:09:33Z")
Hi, Currently my logstash instances are running on native docker. Config and pipeline config are mounted from host filesystem into logstash's container directories. I use only one host in this environment, so mounting f… --- ## [Logstash RabbitMQ input with lazy queue](https://discuss.elastic.co/t/logstash-rabbitmq-input-with-lazy-queue/185221)
**Author:** [@pszemesy](https://discuss.elastic.co/u/pszemesy)\ **Replies:** 1\ **Last updated:** [June 12, 2019, 11:49am UTC](https://discuss.elastic.co/t/logstash-rabbitmq-input-with-lazy-queue/185221 "2019-06-12T11:49:21Z")
Hi, I would like to load data from RabbitMQ with Logstash with the following pipeline: input { rabbitmq { host =\> "xxx.xxx.xxx.xxx" port =\> 5672 user =\> "guest" password =\> "guest" queue =\> "test\_queue" durable … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=528) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=530)