# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=511 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 512 --- ## [Some of my random logs are not parsing from the whole log file.What could be the possible reason for this?](https://discuss.elastic.co/t/some-of-my-random-logs-are-not-parsing-from-the-whole-log-file-what-could-be-the-possible-reason-for-this/189916)
**Author:** [@deepanshu\_goel](https://discuss.elastic.co/u/deepanshu_goel)\ **Replies:** 3\ **Last updated:** [July 12, 2019, 4:42pm UTC](https://discuss.elastic.co/t/some-of-my-random-logs-are-not-parsing-from-the-whole-log-file-what-could-be-the-possible-reason-for-this/189916 "2019-07-12T16:42:55Z")
Every time when i parse my log file there are some random logs which are not being parsed (every time a random set ). --- ## [Cannot able to get the SQLSERVER data into elasticsearch](https://discuss.elastic.co/t/cannot-able-to-get-the-sqlserver-data-into-elasticsearch/190223)
**Author:** [@VIGNESHkumar](https://discuss.elastic.co/u/VIGNESHkumar)\ **Replies:** 1\ **Last updated:** [July 12, 2019, 4:38pm UTC](https://discuss.elastic.co/t/cannot-able-to-get-the-sqlserver-data-into-elasticsearch/190223 "2019-07-12T16:38:20Z")
It shows this error Can anyone please provide suggessions thanks in advance config File input { jdbc { # SqlServer jdbc connection string to your database, productdb # "jdbc:sqlserver://HostName\\instanceName;databa… --- ## [S3 plugin not functioning correctly for GZ files from Firehose](https://discuss.elastic.co/t/s3-plugin-not-functioning-correctly-for-gz-files-from-firehose/190269)
**Author:** [@Arpan\_Patnaik](https://discuss.elastic.co/u/Arpan_Patnaik)\ **Replies:** 0\ **Last updated:** [July 12, 2019, 3:37pm UTC](https://discuss.elastic.co/t/s3-plugin-not-functioning-correctly-for-gz-files-from-firehose/190269 "2019-07-12T15:37:32Z")
Hello! I was testing the s3 plugin for a production POC where a Firehose delivery system is delivering Cloudwatch logs into an S3 bucket from where I am reading it with the S3 plugin into logstash My logstash config is… --- ## [Get filename using beats and filter it in logstash](https://discuss.elastic.co/t/get-filename-using-beats-and-filter-it-in-logstash/190058)
**Author:** [@111175](https://discuss.elastic.co/u/111175)\ **Replies:** 5\ **Last updated:** [July 12, 2019, 2:00pm UTC](https://discuss.elastic.co/t/get-filename-using-beats-and-filter-it-in-logstash/190058 "2019-07-12T14:00:16Z")
Hi, I have logfile with date as it's name and time inside it. I'm using filebeats to get logs from remote server. Is it possible to combine date with time somehow and put in @timestamp? I tried something like this: // … --- ## [Logstash-codec-avro parsing wrong values](https://discuss.elastic.co/t/logstash-codec-avro-parsing-wrong-values/190246)
**Author:** [@Stepan\_Urban](https://discuss.elastic.co/u/Stepan_Urban)\ **Replies:** 0\ **Last updated:** [July 12, 2019, 1:25pm UTC](https://discuss.elastic.co/t/logstash-codec-avro-parsing-wrong-values/190246 "2019-07-12T13:25:04Z")
Hi, I'm writing AVRO data to Kafka and then using logstash to put them into Elasticsearch. But all float values printed by logstash are wrong. To Kafka I'm sending following record ProducerRecord(topic=weather, partiti… --- ## [Logstash not starting after installation and security enabled](https://discuss.elastic.co/t/logstash-not-starting-after-installation-and-security-enabled/189945)
**Author:** [@svm89](https://discuss.elastic.co/u/svm89)\ **Replies:** 1\ **Last updated:** [July 12, 2019, 1:13pm UTC](https://discuss.elastic.co/t/logstash-not-starting-after-installation-and-security-enabled/189945 "2019-07-12T13:13:35Z")
Hi guys, I have installed a new ELK stack platform on the same machine and also enabled the security features (user, roles). The problem I am facing is that Logstash starts and then crashes with the following logs: \[2… --- ## [More consumer threads than the number specified in the input plugin](https://discuss.elastic.co/t/more-consumer-threads-than-the-number-specified-in-the-input-plugin/190241)
**Author:** [@amitavmohanty01](https://discuss.elastic.co/u/amitavmohanty01)\ **Replies:** 0\ **Last updated:** [July 12, 2019, 1:05pm UTC](https://discuss.elastic.co/t/more-consumer-threads-than-the-number-specified-in-the-input-plugin/190241 "2019-07-12T13:05:32Z")
I have 6 logstash nodes. Each node has a pipeline with the following input section. input { kafka { bootstrap\_servers =\> "\" client\_id =\> "unique\_client\_id" group\_id =\> "common\_group\_… --- ## [Probleme of path in logstash file of configuration](https://discuss.elastic.co/t/probleme-of-path-in-logstash-file-of-configuration/190213)
**Author:** [@Moetaz\_Jebri](https://discuss.elastic.co/u/Moetaz_Jebri)\ **Replies:** 0\ **Last updated:** [July 12, 2019, 10:54am UTC](https://discuss.elastic.co/t/probleme-of-path-in-logstash-file-of-configuration/190213 "2019-07-12T10:54:28Z")
My probleme is the path in file configuration the logstash is not reading the all the files in the path his reading them as empty file . this is the path :path =\> "D:/log2/\*.log" --- ## [Logstash config version](https://discuss.elastic.co/t/logstash-config-version/190189)
**Author:** [@smerzlyakov](https://discuss.elastic.co/u/smerzlyakov)\ **Replies:** 0\ **Last updated:** [July 12, 2019, 9:32am UTC](https://discuss.elastic.co/t/logstash-config-version/190189 "2019-07-12T09:32:43Z")
Hello. What is the best way to set logstash config version? When i change config for my custom log source, i must change index name and so on. So, is it some @metadata field, that i can use for setting config version,… --- ## [Indexing Json file](https://discuss.elastic.co/t/indexing-json-file/190183)
**Author:** [@luc1](https://discuss.elastic.co/u/luc1)\ **Replies:** 0\ **Last updated:** [July 12, 2019, 9:10am UTC](https://discuss.elastic.co/t/indexing-json-file/190183 "2019-07-12T09:10:58Z")
Hi, I'm trying to parse JSON file with logstash. In kibana I see the field "counters" of my JSON file like that : What i'm interested in is to get the "tele" in the "sdn" attribute I tried to do a conf file like th… --- ## [JDBC-Input-Plug-In unable to execute SELECT COUTN(\*) FROM sys.session](https://discuss.elastic.co/t/jdbc-input-plug-in-unable-to-execute-select-coutn-from-sys-session/190156)
**Author:** [@Arcefi](https://discuss.elastic.co/u/Arcefi)\ **Replies:** 0\ **Last updated:** [July 12, 2019, 7:55am UTC](https://discuss.elastic.co/t/jdbc-input-plug-in-unable-to-execute-select-coutn-from-sys-session/190156 "2019-07-12T07:55:23Z")
I want to use Logstash to get the current count of all sessions on my MySQL DB every second. Thats my configuration for the pipeline: input { jdbc { jdbc\_driver\_library =\> "/usr/share/java/mysql-connector-java-5.… --- ## [Kafka input to logstash](https://discuss.elastic.co/t/kafka-input-to-logstash/190150)
**Author:** [@Harsha606](https://discuss.elastic.co/u/Harsha606)\ **Replies:** 0\ **Last updated:** [July 12, 2019, 7:07am UTC](https://discuss.elastic.co/t/kafka-input-to-logstash/190150 "2019-07-12T07:07:16Z")
Hi All, i am trying to add multiple kafka topics to logstash and output to elastic. my requirement here is, the index name should be the topic name, as below... input { kafka { bootstrap\_servers =\> "broker.kafka-… --- ## [Logstash multi pipeline can´t connect to one elasticsearch node](https://discuss.elastic.co/t/logstash-multi-pipeline-can-t-connect-to-one-elasticsearch-node/189937)
**Author:** [@Arcefi](https://discuss.elastic.co/u/Arcefi)\ **Replies:** 2\ **Last updated:** [July 12, 2019, 6:42am UTC](https://discuss.elastic.co/t/logstash-multi-pipeline-can-t-connect-to-one-elasticsearch-node/189937 "2019-07-12T06:42:52Z")
Im trying to set up a complete kpi preformance tracking for my system. At the moment im using elasticsearch 7.1, packetbeat 7.1 logstash 7.1 and a MySQL DB. My tracking includes following components: Elasticsearch re… --- ## [Logstash parse data deeply](https://discuss.elastic.co/t/logstash-parse-data-deeply/190146)
**Author:** [@Ahmet\_Kartal](https://discuss.elastic.co/u/Ahmet_Kartal)\ **Replies:** 0\ **Last updated:** [July 12, 2019, 6:09am UTC](https://discuss.elastic.co/t/logstash-parse-data-deeply/190146 "2019-07-12T06:09:00Z")
Hello, I have data that I want to parse. I have splitted some part of my data but I need to parse it again. Is there any other way to parse my data? This is my data: IP Adress - - \[01/Jul/2019:13:03:07 +0300\] "POST /sea… --- ## [Logstash plugin couchdb\_changes integration getting error](https://discuss.elastic.co/t/logstash-plugin-couchdb-changes-integration-getting-error/190138)
**Author:** [@ranjan](https://discuss.elastic.co/u/ranjan)\ **Replies:** 0\ **Last updated:** [July 12, 2019, 4:41am UTC](https://discuss.elastic.co/t/logstash-plugin-couchdb-changes-integration-getting-error/190138 "2019-07-12T04:41:12Z")
i am getting error while integrating couchdb databse with logstash i.e. forwarded to elasticsearch database and kibana. my config file for logstash is couchdb\_changes { db =\> "demo" #type =\> "test" #host =\>… --- ## [Date parse failure but format looks correct?](https://discuss.elastic.co/t/date-parse-failure-but-format-looks-correct/189693)
**Author:** [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Replies:** 3\ **Last updated:** [July 11, 2019, 11:35pm UTC](https://discuss.elastic.co/t/date-parse-failure-but-format-looks-correct/189693 "2019-07-11T23:35:44Z")
Hi, I'm using a bit of ruby to generate a future timestamp. I want to set this as the @timestamp but for whatever reason the date is not accepted no matter the date format I pick (ISO8601, mm:ss,SSS etc.). stdout @cur… --- ## [Ruby set first day of next month?](https://discuss.elastic.co/t/ruby-set-first-day-of-next-month/189654)
**Author:** [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Replies:** 2\ **Last updated:** [July 11, 2019, 11:35pm UTC](https://discuss.elastic.co/t/ruby-set-first-day-of-next-month/189654 "2019-07-11T23:35:01Z")
Hi, Is it possible to get the first day of the next month using ruby? I need to to update some documents with a future date in a format such as 2019/07/01. I tried using date and today but none of them seem to work. Lo… --- ## [How to mutate %{\[host\]\[hostname\]} to lowercase for pipeline output](https://discuss.elastic.co/t/how-to-mutate-host-hostname-to-lowercase-for-pipeline-output/190116)
**Author:** [@Thanura\_Kannangara](https://discuss.elastic.co/u/Thanura_Kannangara)\ **Replies:** 1\ **Last updated:** [July 11, 2019, 10:48pm UTC](https://discuss.elastic.co/t/how-to-mutate-host-hostname-to-lowercase-for-pipeline-output/190116 "2019-07-11T22:48:32Z")
I wanted to create indices with hostnames EG: logstash-\-7.2.0-2019.07.11 My output for the pipeline looks like this output { elasticsearch { ssl =\> true hosts =\> \["xxxxxxx.us-east-1.aws.… --- ## [Kafka INVALID\_FETCH\_SESSION\_EPOCH](https://discuss.elastic.co/t/kafka-invalid-fetch-session-epoch/187441)
**Author:** [@Chris\_Denneen](https://discuss.elastic.co/u/Chris_Denneen)\ **Replies:** 2\ **Last updated:** [July 11, 2019, 6:46pm UTC](https://discuss.elastic.co/t/kafka-invalid-fetch-session-epoch/187441 "2019-07-11T18:46:26Z")
It appears this has been asked more than once and never answered... anyone from @elastic @logstash able to help? {"level":"INFO","loggerName":"org.apache.kafka.clients.FetchSessionHandler","timeMillis":1561500837923… --- ## [Need help deciding which input plugin to use](https://discuss.elastic.co/t/need-help-deciding-which-input-plugin-to-use/188360)
**Author:** [@az123](https://discuss.elastic.co/u/az123)\ **Replies:** 5\ **Last updated:** [July 11, 2019, 4:47pm UTC](https://discuss.elastic.co/t/need-help-deciding-which-input-plugin-to-use/188360 "2019-07-11T16:47:21Z")
I want to use log stash to act as a tcp client to get some data from the server. I need logstash to send a json message over to the server to request the data of interest before the server can send that data over (More l… --- ## [Grok - Integer data not being passed into fields when using :int](https://discuss.elastic.co/t/grok-integer-data-not-being-passed-into-fields-when-using-int/189877)
**Author:** [@anda](https://discuss.elastic.co/u/anda)\ **Replies:** 2\ **Last updated:** [July 11, 2019, 4:02pm UTC](https://discuss.elastic.co/t/grok-integer-data-not-being-passed-into-fields-when-using-int/189877 "2019-07-11T16:02:33Z")
I've been using https://grokdebug.herokuapp.com/ to debug and learn grok patterns. Input: 10-07-19 14:06:00.025 SomeString 2963867553 3028425000 205809770 209513296 2298 2292 8 37 90 1562792747810 1562791897820 1562792… --- ## [Using 2 .csv inputs](https://discuss.elastic.co/t/using-2-csv-inputs/190044)
**Author:** [@yess](https://discuss.elastic.co/u/yess)\ **Replies:** 3\ **Last updated:** [July 11, 2019, 3:34pm UTC](https://discuss.elastic.co/t/using-2-csv-inputs/190044 "2019-07-11T15:34:33Z")
I am trying to use 2 folders with .csv logs in as the input to my .conf file, yet it doesn't seem to work: here is my code input{ file { type =\> "file1" path =\> "/place/holder/place/holder/\*.csv" start\_posit… --- ## [Ingest multiple json files under a folder](https://discuss.elastic.co/t/ingest-multiple-json-files-under-a-folder/190026)
**Author:** [@decodeit](https://discuss.elastic.co/u/decodeit)\ **Replies:** 10\ **Last updated:** [July 11, 2019, 2:32pm UTC](https://discuss.elastic.co/t/ingest-multiple-json-files-under-a-folder/190026 "2019-07-11T14:32:08Z")
I am trying to make logstash ingest multiple json files under a folder. Each json file is multiline about 100 line. I don't see any out put . here is my config. \`input { file{ path =\> "/data/nhs.json" start\_posi… --- ## [Parsing of a sub-element](https://discuss.elastic.co/t/parsing-of-a-sub-element/190028)
**Author:** [@minder](https://discuss.elastic.co/u/minder)\ **Replies:** 2\ **Last updated:** [July 11, 2019, 2:31pm UTC](https://discuss.elastic.co/t/parsing-of-a-sub-element/190028 "2019-07-11T14:31:03Z")
Hi, I've got an input like this: { "\_index": "index-2019.07.11", "\_type": "doc", "\_id": "id", "\_version": 1, "\_score": null, "\_source": { "message": "{ "time": "2019-07-11T14:17:18.7132501+03:00", "level": "ERRO… --- ## [Field spilt](https://discuss.elastic.co/t/field-spilt/189694)
**Author:** [@galtryn](https://discuss.elastic.co/u/galtryn)\ **Replies:** 12\ **Last updated:** [July 11, 2019, 1:24pm UTC](https://discuss.elastic.co/t/field-spilt/189694 "2019-07-11T13:24:10Z")
Hi Guys, I am ingesting some IoT sensor data into Elasticsearch via logstash, the payload of the message is a HEX string which currently looks something like this; 129, 91, 0, 0, 28, 28, 29, 0, 4, 0, 0, 0, 11, 38, 0, 2… --- ## [HTTP Output Failure - Unable to send logstash output to elasticsearch](https://discuss.elastic.co/t/http-output-failure-unable-to-send-logstash-output-to-elasticsearch/189974)
**Author:** [@Vivek\_Sharma](https://discuss.elastic.co/u/Vivek_Sharma)\ **Replies:** 1\ **Last updated:** [July 11, 2019, 1:13pm UTC](https://discuss.elastic.co/t/http-output-failure-unable-to-send-logstash-output-to-elasticsearch/189974 "2019-07-11T13:13:30Z")
Hi I am trying to parse a text file containing 3 fields and storing the result in elasticsearch using http. Below is the code I have created for this. input { beats { port =\> 5044 } } filter { dissect { … --- ## [ArcSight module CEF input parsing (over TCP) \[SOLVED\]](https://discuss.elastic.co/t/arcsight-module-cef-input-parsing-over-tcp-solved/190018)
**Author:** [@falken](https://discuss.elastic.co/u/falken)\ **Replies:** 0\ **Last updated:** [July 11, 2019, 1:06pm UTC](https://discuss.elastic.co/t/arcsight-module-cef-input-parsing-over-tcp-solved/190018 "2019-07-11T13:06:34Z")
Good afternoon folks! I do not know if what I had found works for myself (only) is something really useful for everyone, or it just dipends from the particular environment I'm working in. Anyway, I'm going to share it wi… --- ## [Windows -](https://discuss.elastic.co/t/windows/189887)
**Author:** [@OpSecMonkey](https://discuss.elastic.co/u/OpSecMonkey)\ **Replies:** 1\ **Last updated:** [July 11, 2019, 1:01pm UTC](https://discuss.elastic.co/t/windows/189887 "2019-07-11T13:01:13Z")
Good afternoon everyone. So I am running ELK on windows and I am getting this error. Reached open files limit: 4095, set by the 'max\_open\_files' option or default, files yet to open: XXXXXX Its still moving along its … --- ## [Create a new field type date from csv with logstash](https://discuss.elastic.co/t/create-a-new-field-type-date-from-csv-with-logstash/189940)
**Author:** [@GitsBdr](https://discuss.elastic.co/u/GitsBdr)\ **Replies:** 2\ **Last updated:** [July 11, 2019, 12:44pm UTC](https://discuss.elastic.co/t/create-a-new-field-type-date-from-csv-with-logstash/189940 "2019-07-11T12:44:14Z")
Hi all, Thanks in advance for your help here Just trying to figure out what I am doing wrong. I've got a field from a csv file called 'time' and looks like : '07:00:01.000' and I'm trying to create the field 'eventdat… --- ## [How do i match in grok](https://discuss.elastic.co/t/how-do-i-match-in-grok/189995)
**Author:** [@mancharagopan](https://discuss.elastic.co/u/mancharagopan)\ **Replies:** 4\ **Last updated:** [July 11, 2019, 12:34pm UTC](https://discuss.elastic.co/t/how-do-i-match-in-grok/189995 "2019-07-11T12:34:23Z")
I want to match ASA syslog message for message id, severity and add it to separate fields. How do i match it in grok? \<166\>Jul 11 2019 16:46:45: %ASA-6-113012: AAA user authentication Successful : local database : user … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=510) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=512)