# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1089 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1090 --- ## [Logstash Sending data into wrong/another document type instead of document on which i have configured it](https://discuss.elastic.co/t/logstash-sending-data-into-wrong-another-document-type-instead-of-document-on-which-i-have-configured-it/63568)
**Author:** [@Tushar\_Sharma1](https://discuss.elastic.co/u/Tushar_Sharma1)\ **Replies:** 3\ **Last updated:** [October 21, 2016, 7:51am UTC](https://discuss.elastic.co/t/logstash-sending-data-into-wrong-another-document-type-instead-of-document-on-which-i-have-configured-it/63568 "2016-10-21T07:51:49Z")
Hi All, I have written filter in logstash and placed them in logstash conf.d directory and i have set same index but different document type for elasticsearch output of each filter.But when I have run query in elasticsea… --- ## [Combine strings from array into field](https://discuss.elastic.co/t/combine-strings-from-array-into-field/63520)
**Author:** [@rsaeks](https://discuss.elastic.co/u/rsaeks)\ **Replies:** 2\ **Last updated:** [October 21, 2016, 5:54am UTC](https://discuss.elastic.co/t/combine-strings-from-array-into-field/63520 "2016-10-21T05:54:53Z")
Hi all, I'm working with manipulating some data from a logstash filter and haven't had any luck in figuring out how to do one specific action. I'm parsing a string and am pulling all the needed data where in my field c… --- ## [Logstash to datadog\_metrics - example needed](https://discuss.elastic.co/t/logstash-to-datadog-metrics-example-needed/63564)
**Author:** [@gmmurugan](https://discuss.elastic.co/u/gmmurugan)\ **Replies:** 0\ **Last updated:** [October 21, 2016, 4:49am UTC](https://discuss.elastic.co/t/logstash-to-datadog-metrics-example-needed/63564 "2016-10-21T04:49:09Z")
datadog\_metrics { api\_key =\> "xx" metric\_name =\> "unresolved\_counts" metric\_value=\> "50" } i would like to get some examples to how we can send kvp info to datadog.. is api\_key is enough?? --- ## [Reasoning behind a daily logstash index?](https://discuss.elastic.co/t/reasoning-behind-a-daily-logstash-index/62769)
**Author:** [@chrisan](https://discuss.elastic.co/u/chrisan)\ **Replies:** 2\ **Last updated:** [October 20, 2016, 10:20pm UTC](https://discuss.elastic.co/t/reasoning-behind-a-daily-logstash-index/62769 "2016-10-20T22:20:15Z")
Hello, I'm still coming up to speed with ELK. My use case is to analyze AWS CloudFront logs in aggregates (such as total bandwidth for a folder in a bucket per month). These logs are about 5,000 KB compressed per day … --- ## [Logstash Unable to parse dot inside a json](https://discuss.elastic.co/t/logstash-unable-to-parse-dot-inside-a-json/63294)
**Author:** [@javatechy](https://discuss.elastic.co/u/javatechy)\ **Replies:** 1\ **Last updated:** [October 20, 2016, 7:48pm UTC](https://discuss.elastic.co/t/logstash-unable-to-parse-dot-inside-a-json/63294 "2016-10-20T19:48:09Z")
I am trying to parse my CSV fomatted logs using logstash. I am facing problem in parsing a json which contains dot(.) in the key of json. I took reference from this link ref but didn't work for me My sample input is … --- ## [Grok pattern to read the testname in a multiline logfile](https://discuss.elastic.co/t/grok-pattern-to-read-the-testname-in-a-multiline-logfile/63248)
**Author:** [@sharath3185](https://discuss.elastic.co/u/sharath3185)\ **Replies:** 1\ **Last updated:** [October 20, 2016, 7:45pm UTC](https://discuss.elastic.co/t/grok-pattern-to-read-the-testname-in-a-multiline-logfile/63248 "2016-10-20T19:45:23Z")
Hi I am trying to fetch the testnames from a multiline logfile using logstash. Below is the sample: testname1.sh DH no TIMESTAMP 1454774186 LASTSTATION xxx LASTPASS 1454774186 PASSNUM … --- ## [Problem when try install logstash jdbc input plugin in ubuntu](https://discuss.elastic.co/t/problem-when-try-install-logstash-jdbc-input-plugin-in-ubuntu/63530)
**Author:** [@renan.azevedo](https://discuss.elastic.co/u/renan.azevedo)\ **Replies:** 0\ **Last updated:** [October 20, 2016, 5:25pm UTC](https://discuss.elastic.co/t/problem-when-try-install-logstash-jdbc-input-plugin-in-ubuntu/63530 "2016-10-20T17:25:26Z")
Hi, I can´t install logstash in ubuntu server 14.04. I install jruby and bundler gem. Clone the folder of Logstash JDBC Input Plugin. But when I run "bundle install" I get this error: Could not find gem 'logsta… --- ## [Not able to read Kafka - Avro Schema messages](https://discuss.elastic.co/t/not-able-to-read-kafka-avro-schema-messages/24979)
**Author:** [@karthikr](https://discuss.elastic.co/u/karthikr)\ **Replies:** 15\ **Last updated:** [October 20, 2016, 3:58pm UTC](https://discuss.elastic.co/t/not-able-to-read-kafka-avro-schema-messages/24979 "2016-10-20T15:58:29Z")
I want to read snappy compressed avro messages stored in kafka into elastic. I am using the Kafka Input Plugin and the Avro Logstash Codec but not able to read the compressed messages. Do note that I am able to read plai… --- ## [Install Cassandra Driver (Ruby) for Logstash](https://discuss.elastic.co/t/install-cassandra-driver-ruby-for-logstash/63490)
**Author:** [@TWalter](https://discuss.elastic.co/u/TWalter)\ **Replies:** 0\ **Last updated:** [October 20, 2016, 10:00am UTC](https://discuss.elastic.co/t/install-cassandra-driver-ruby-for-logstash/63490 "2016-10-20T10:00:53Z")
Hello there, If you want to connect to a cassandra database with the ruby-filter, you can do it the following way. All you need for this, is the datastax cassandra ruby driver: https://github.com/datastax/ruby-driv… --- ## [Not able to parse fields with logstash](https://discuss.elastic.co/t/not-able-to-parse-fields-with-logstash/63468)
**Author:** [@VijayKarthikeyan](https://discuss.elastic.co/u/VijayKarthikeyan)\ **Replies:** 3\ **Last updated:** [October 20, 2016, 11:25am UTC](https://discuss.elastic.co/t/not-able-to-parse-fields-with-logstash/63468 "2016-10-20T11:25:33Z")
Hello everyone, Am using Logstash and am a new user. I have a log file which I need to parse and get some meaningful fields like Timestamp, User, Domain, message, response code, error code etc., I use Grok pattern to p… --- ## [Does Logstash cache the Query Results (JDBC-Input)](https://discuss.elastic.co/t/does-logstash-cache-the-query-results-jdbc-input/63312)
**Author:** [@TWalter](https://discuss.elastic.co/u/TWalter)\ **Replies:** 2\ **Last updated:** [October 20, 2016, 10:03am UTC](https://discuss.elastic.co/t/does-logstash-cache-the-query-results-jdbc-input/63312 "2016-10-20T10:03:37Z")
Hello there, i have an important question about the processes in Logstash. If i Query from a relational Database except 5000 entries. Does Logstash cache the 5000 an executes the filter and output like in a for-loop… --- ## [Integer type question](https://discuss.elastic.co/t/integer-type-question/63309)
**Author:** [@Lego](https://discuss.elastic.co/u/Lego)\ **Replies:** 4\ **Last updated:** [October 20, 2016, 9:51am UTC](https://discuss.elastic.co/t/integer-type-question/63309 "2016-10-20T09:51:17Z")
Hello. When I try to convert string data to integer I can see some things which are not clear for me. For example when value 7 is converted from string to integer it displays as number seven. And it's ok. But whe… --- ## [Parsing different syslog date formats](https://discuss.elastic.co/t/parsing-different-syslog-date-formats/63431)
**Author:** [@pixelrebel](https://discuss.elastic.co/u/pixelrebel)\ **Replies:** 4\ **Last updated:** [October 20, 2016, 7:13am UTC](https://discuss.elastic.co/t/parsing-different-syslog-date-formats/63431 "2016-10-20T07:13:21Z")
I have two types of timestamps coming into my logstash syslog input: SYSLOGTIMESTAMP - "Oct 19 11:29:00" TIMESTAMP\_ISO8601 - "2016-10-19T18:31:52.519Z" My grok below works for both: grok { … --- ## [Failed to parse multiple date fields](https://discuss.elastic.co/t/failed-to-parse-multiple-date-fields/63383)
**Author:** [@setlem](https://discuss.elastic.co/u/setlem)\ **Replies:** 6\ **Last updated:** [October 20, 2016, 6:21am UTC](https://discuss.elastic.co/t/failed-to-parse-multiple-date-fields/63383 "2016-10-20T06:21:59Z")
Hi, my config file is date is parsing only for LAST\_MODIFIED\_DATE. But not for CREATION\_DATE and SUBMITTED\_DATE Suggest me the way to fix this. Thanks!!!!! --- ## [Logstash json to kv output to syslog for splunkindex](https://discuss.elastic.co/t/logstash-json-to-kv-output-to-syslog-for-splunkindex/63452)
**Author:** [@gmmurugan](https://discuss.elastic.co/u/gmmurugan)\ **Replies:** 2\ **Last updated:** [October 20, 2016, 3:38am UTC](https://discuss.elastic.co/t/logstash-json-to-kv-output-to-syslog-for-splunkindex/63452 "2016-10-20T03:38:42Z")
I would like to get data from database and add few fields with builtin json to kv conversion and pusing it to syslog/tcp of splunk indexer. any sample config you can guide? 1. json to kv - any example? 2. add few fiel… --- ## [Update nested object using script with params in Logstash Elasticsearch output](https://discuss.elastic.co/t/update-nested-object-using-script-with-params-in-logstash-elasticsearch-output/49983)
**Author:** [@LetMeR00t](https://discuss.elastic.co/u/LetMeR00t)\ **Replies:** 14\ **Last updated:** [October 20, 2016, 2:30am UTC](https://discuss.elastic.co/t/update-nested-object-using-script-with-params-in-logstash-elasticsearch-output/49983 "2016-10-20T02:30:08Z")
Hello everyone, I would like to update a document that contains a nested object. I've already seen that sounds possible using script. However, it appears that we can't use params in Logstash ... My mapping for the co… --- ## [Issues processing text file with Logstash 2.4](https://discuss.elastic.co/t/issues-processing-text-file-with-logstash-2-4/63446)
**Author:** [@tpnoonan](https://discuss.elastic.co/u/tpnoonan)\ **Replies:** 2\ **Last updated:** [October 19, 2016, 9:39pm UTC](https://discuss.elastic.co/t/issues-processing-text-file-with-logstash-2-4/63446 "2016-10-19T21:39:22Z")
I've got a text file (json) that I want to process and eventually pump into ElasticSearch. I first started with a more elaborate configuration file and have now scaled back to a simple config that uses stdin for input a… --- ## [How to define Kibana/Logstash password encrypted](https://discuss.elastic.co/t/how-to-define-kibana-logstash-password-encrypted/42850)
**Author:** [@Emmanuel\_DEMEY](https://discuss.elastic.co/u/Emmanuel_DEMEY)\ **Replies:** 7\ **Last updated:** [October 19, 2016, 9:27pm UTC](https://discuss.elastic.co/t/how-to-define-kibana-logstash-password-encrypted/42850 "2016-10-19T21:27:20Z")
Hi everyone, I am working with Shield, and I have a question about the configuration of the Kibana or Logstash user in their corresponding configuration file. Is it possible to encrypt password for thes users ? in … --- ## [How can i keep the fields case sensitive using logstash jdbc](https://discuss.elastic.co/t/how-can-i-keep-the-fields-case-sensitive-using-logstash-jdbc/62957)
**Author:** [@saisn](https://discuss.elastic.co/u/saisn)\ **Replies:** 3\ **Last updated:** [October 19, 2016, 6:28pm UTC](https://discuss.elastic.co/t/how-can-i-keep-the-fields-case-sensitive-using-logstash-jdbc/62957 "2016-10-19T18:28:11Z")
Hi , I want the field name to be exactly same when importing from logstash instead of default lowercase conversion For eg: 'FirstName' field in SQL should be imported exactly as 'FirstName' in elasticsearch not 'firs… --- ## [Logstash 2.3.4 as Windows Service using Apache procrun](https://discuss.elastic.co/t/logstash-2-3-4-as-windows-service-using-apache-procrun/62866)
**Author:** [@mdconner](https://discuss.elastic.co/u/mdconner)\ **Replies:** 1\ **Last updated:** [October 19, 2016, 4:13pm UTC](https://discuss.elastic.co/t/logstash-2-3-4-as-windows-service-using-apache-procrun/62866 "2016-10-19T16:13:50Z")
I'm getting the following errors when running "logstash.bat agent -f C:\\temp\\scribe\\data\\logstash\\config --debug": /usr/bin/stty: standard input: Inappropriate ioctl for device io/console not supported; tty will not be… --- ## [Geo mapping issue](https://discuss.elastic.co/t/geo-mapping-issue/63385)
**Author:** [@Raj\_Kumar](https://discuss.elastic.co/u/Raj_Kumar)\ **Replies:** 3\ **Last updated:** [October 19, 2016, 1:07pm UTC](https://discuss.elastic.co/t/geo-mapping-issue/63385 "2016-10-19T13:07:07Z")
Hi All, I have a issues in mapping the geo ip location to geo point.Could anyone please help me on this.Iam trying to analyse our cisco netflow data .When i tried to create tile map its resulting in no results found. … --- ## [If value in log string doesnot exist](https://discuss.elastic.co/t/if-value-in-log-string-doesnot-exist/63381)
**Author:** [@Artyom\_Davydov](https://discuss.elastic.co/u/Artyom_Davydov)\ **Replies:** 3\ **Last updated:** [October 19, 2016, 1:03pm UTC](https://discuss.elastic.co/t/if-value-in-log-string-doesnot-exist/63381 "2016-10-19T13:03:57Z")
Hi all. I have this log: Oct 19 13:19:22 prod-mail amavis\[2815\]: (02815-17) Blocked SPAM {DiscardedInternal}, LOCAL \[63.240.118.168\]:41519 \[63.240.118.168\] some@mail.be -\> some2@mail.ru, Queue-ID: 621CD1E07E7, Message… --- ## [Mutate only matching](https://discuss.elastic.co/t/mutate-only-matching/63320)
**Author:** [@emanuelef](https://discuss.elastic.co/u/emanuelef)\ **Replies:** 5\ **Last updated:** [October 19, 2016, 11:09am UTC](https://discuss.elastic.co/t/mutate-only-matching/63320 "2016-10-19T11:09:51Z")
Hi, I'd like to "extract" some fields from a specific kind of log and convert them to the right type. I can do that when all logs pass the match but in this case the logs I'm getting have different formats. So I would li… --- ## [Changing file content during indexation](https://discuss.elastic.co/t/changing-file-content-during-indexation/63032)
**Author:** [@frederes](https://discuss.elastic.co/u/frederes)\ **Replies:** 4\ **Last updated:** [October 19, 2016, 8:09am UTC](https://discuss.elastic.co/t/changing-file-content-during-indexation/63032 "2016-10-19T08:09:52Z")
I have a logstah monitoring all files of type \*.csv in a directory. If i delete or change the content of one .csv (remove lines), will logstash be perturbated in its indexation task ? --- ## [Logstash not picking older logs](https://discuss.elastic.co/t/logstash-not-picking-older-logs/62944)
**Author:** [@VijayKarthikeyan](https://discuss.elastic.co/u/VijayKarthikeyan)\ **Replies:** 6\ **Last updated:** [October 19, 2016, 7:21am UTC](https://discuss.elastic.co/t/logstash-not-picking-older-logs/62944 "2016-10-19T07:21:10Z")
I have a scenario where I need to pick 4 to 5 years older data using Logstash. Below is the config file input { file { path =\> \[ "cricsheet-xml-master\\0.7\\5\*.xml" \] type =\> "CricketData" start\_position =\> … --- ## [Rsyslogd - Filebeat - Logstash =\> bad severity/priority](https://discuss.elastic.co/t/rsyslogd-filebeat-logstash-bad-severity-priority/63303)
**Author:** [@smutel](https://discuss.elastic.co/u/smutel)\ **Replies:** 1\ **Last updated:** [October 19, 2016, 7:08am UTC](https://discuss.elastic.co/t/rsyslogd-filebeat-logstash-bad-severity-priority/63303 "2016-10-19T07:08:21Z")
Hello, I think that this question has been already asked a lot but i didn't find any answer ... Severity of all my syslog logs is notice and priority of all my syslog logs is user-level. I tried to change the rsyslog … --- ## [Shipping and indexing](https://discuss.elastic.co/t/shipping-and-indexing/63161)
**Author:** [@laywin](https://discuss.elastic.co/u/laywin)\ **Replies:** 4\ **Last updated:** [October 19, 2016, 6:58am UTC](https://discuss.elastic.co/t/shipping-and-indexing/63161 "2016-10-19T06:58:48Z")
i read the reference https://www.elastic.co/guide/en/logstash/current/deploying-and-scaling.html, in the article,shipping logstash collect logs first then logs pass messge queue finally indexing logstash filter logs. … --- ## [Logstash Pipeline aborted due to error](https://discuss.elastic.co/t/logstash-pipeline-aborted-due-to-error/63177)
**Author:** [@bopa](https://discuss.elastic.co/u/bopa)\ **Replies:** 2\ **Last updated:** [October 19, 2016, 6:40am UTC](https://discuss.elastic.co/t/logstash-pipeline-aborted-due-to-error/63177 "2016-10-19T06:40:28Z")
following is my logstash config file . configtest shows OK but I'm getting "Pipeline aborted due to error" in logstash logs. logstash config input { tcp { port =\> 5000 type =\> 'eventlog' … --- ## [Grok filter needed after filebeat send data](https://discuss.elastic.co/t/grok-filter-needed-after-filebeat-send-data/63329)
**Author:** [@cetanhota](https://discuss.elastic.co/u/cetanhota)\ **Replies:** 1\ **Last updated:** [October 19, 2016, 6:00am UTC](https://discuss.elastic.co/t/grok-filter-needed-after-filebeat-send-data/63329 "2016-10-19T06:00:02Z")
Hello all new at ELK. Been doing well so far, but I have come across 1 problem and I figured it was worth asking. I have filebeat sending me data off several servers. Filebeat is working well and I am able to filter o… --- ## [Need Help for grok pattern for logstash-2.1.0](https://discuss.elastic.co/t/need-help-for-grok-pattern-for-logstash-2-1-0/63019)
**Author:** [@Roshan](https://discuss.elastic.co/u/Roshan)\ **Replies:** 1\ **Last updated:** [October 19, 2016, 5:43am UTC](https://discuss.elastic.co/t/need-help-for-grok-pattern-for-logstash-2-1-0/63019 "2016-10-19T05:43:00Z")
I've been building some grok patterns to parse the log file and everything has been working fine. I create the grok patterns at http://grokdebug.herokuapp.com/ and sites show the patterns matching perfectly. I'm using … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1088) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1090)