# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=415 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 416 --- ## [Need title/name of file for logstash ingestion](https://discuss.elastic.co/t/need-title-name-of-file-for-logstash-ingestion/211308)
**Author:** [@edster](https://discuss.elastic.co/u/edster)\ **Replies:** 11\ **Last updated:** [January 22, 2020, 9:22pm UTC](https://discuss.elastic.co/t/need-title-name-of-file-for-logstash-ingestion/211308 "2020-01-22T21:22:13Z")
Hello, I am trying to get part of the file names for ingestion for each record, however, I am ingesting multiple files using "cat /filespath/ | /logstashpath/ -f /logstashconfig/" command. The reason i am running it lik… --- ## [Logstash Cloudwatch logs collect based on timestamp](https://discuss.elastic.co/t/logstash-cloudwatch-logs-collect-based-on-timestamp/216375)
**Author:** [@Rajesh\_Somasundaram](https://discuss.elastic.co/u/Rajesh_Somasundaram)\ **Replies:** 0\ **Last updated:** [January 24, 2020, 7:13am UTC](https://discuss.elastic.co/t/logstash-cloudwatch-logs-collect-based-on-timestamp/216375 "2020-01-24T07:13:25Z")
Hi Team, We have AWS cloud watch which has 3+ years of logs stored in it. I would like to collect the logs from AWS cloud watch and publish it to elastic search only for the last two weeks data since we really dont anal… --- ## [Remote logstash server](https://discuss.elastic.co/t/remote-logstash-server/216371)
**Author:** [@Ankit\_Sawant](https://discuss.elastic.co/u/Ankit_Sawant)\ **Replies:** 0\ **Last updated:** [January 24, 2020, 6:53am UTC](https://discuss.elastic.co/t/remote-logstash-server/216371 "2020-01-24T06:53:48Z")
Hello everyone, I am using logstash on a linux based server . I am collecting the logs on this server and after processing the logs with logstash i want to send these logs to elastic search which is on remote server .fo… --- ## [Input Elastic Plugin in Logstash Error](https://discuss.elastic.co/t/input-elastic-plugin-in-logstash-error/216188)
**Author:** [@reyhanadp](https://discuss.elastic.co/u/reyhanadp)\ **Replies:** 2\ **Last updated:** [January 24, 2020, 1:01am UTC](https://discuss.elastic.co/t/input-elastic-plugin-in-logstash-error/216188 "2020-01-24T01:01:42Z")
Hello Everyone, I want to retrieve data from elastic using logstash. the data that I want to get is sorted by the value column. when I retrieve data from elastic without setting a schedule, the results of data output ar… --- ## [Logstash Parsing File (Without Filebeat)](https://discuss.elastic.co/t/logstash-parsing-file-without-filebeat/216160)
**Author:** [@ellje](https://discuss.elastic.co/u/ellje)\ **Replies:** 8\ **Last updated:** [January 24, 2020, 12:34am UTC](https://discuss.elastic.co/t/logstash-parsing-file-without-filebeat/216160 "2020-01-24T00:34:22Z")
Trying to simply have logstash parse a file. I've checked that the file input path is correct, and when I tail the logtash log file, there is no update/activity. Is there anything else that I can check to ensure that it'… --- ## [Insert all records of a table in mysql with no unique column in elastic without repitition](https://discuss.elastic.co/t/insert-all-records-of-a-table-in-mysql-with-no-unique-column-in-elastic-without-repitition/215377)
**Author:** [@RITZ\_VERMA](https://discuss.elastic.co/u/RITZ_VERMA)\ **Replies:** 2\ **Last updated:** [January 23, 2020, 6:44pm UTC](https://discuss.elastic.co/t/insert-all-records-of-a-table-in-mysql-with-no-unique-column-in-elastic-without-repitition/215377 "2020-01-23T18:44:13Z")
I want to send data from mysql table to elastic search .I am currently using jdbc plugin but it keep inserting same records again .I have no unique column in my mysql table.is there a way to send the records exactly once … --- ## [Single data source split to different indexes](https://discuss.elastic.co/t/single-data-source-split-to-different-indexes/215695)
**Author:** [@BeMoore](https://discuss.elastic.co/u/BeMoore)\ **Replies:** 11\ **Last updated:** [January 23, 2020, 2:57pm UTC](https://discuss.elastic.co/t/single-data-source-split-to-different-indexes/215695 "2020-01-23T14:57:20Z")
Morning all ! i have a question, if i have a json data source like : { "log\_url" =\> "http://127.0.0.1/log.txt", "key" =\> "22op3dfe", "raw\_msg" =\> "404.19 – Denied by filtering rule", "MD5" =\> "2c5c… --- ## [Logstash jdbc input plugin pagination not working on version 7.4.2 Replay](https://discuss.elastic.co/t/logstash-jdbc-input-plugin-pagination-not-working-on-version-7-4-2-replay/216261)
**Author:** [@taku333](https://discuss.elastic.co/u/taku333)\ **Replies:** 0\ **Last updated:** [January 23, 2020, 2:11pm UTC](https://discuss.elastic.co/t/logstash-jdbc-input-plugin-pagination-not-working-on-version-7-4-2-replay/216261 "2020-01-23T14:11:49Z")
@Rafa About the following URL Isn't this the answer? --- ## [Logstash processing twice then 'block\_in\_start\_workers'](https://discuss.elastic.co/t/logstash-processing-twice-then-block-in-start-workers/216080)
**Author:** [@guctum](https://discuss.elastic.co/u/guctum)\ **Replies:** 2\ **Last updated:** [January 23, 2020, 1:58pm UTC](https://discuss.elastic.co/t/logstash-processing-twice-then-block-in-start-workers/216080 "2020-01-23T13:58:33Z")
Using Logstash to get information from Oracle Enterprise Manager, keep getting an error relating to multi\_recieve with 'block in start\_workers' repeating over and over whenever Logstash tries again. I have the below conf… --- ## [Wrong argument type string expected Timestamp](https://discuss.elastic.co/t/wrong-argument-type-string-expected-timestamp/216229)
**Author:** [@morpheus](https://discuss.elastic.co/u/morpheus)\ **Replies:** 1\ **Last updated:** [January 23, 2020, 1:53pm UTC](https://discuss.elastic.co/t/wrong-argument-type-string-expected-timestamp/216229 "2020-01-23T13:53:51Z")
Greetings, I have seen other similar threads to this but not able to solve my problem, I am getting the error: \[2020-01-22T10:45:38,409\]\[WARN \]\[logstash.inputs.gelf \] Gelf (tcp): client socket failed. {:exception=\>… --- ## [Missing log lines filebeat to logstash](https://discuss.elastic.co/t/missing-log-lines-filebeat-to-logstash/216255)
**Author:** [@danielwind83](https://discuss.elastic.co/u/danielwind83)\ **Replies:** 0\ **Last updated:** [January 23, 2020, 1:46pm UTC](https://discuss.elastic.co/t/missing-log-lines-filebeat-to-logstash/216255 "2020-01-23T13:46:37Z")
Hi All, We are running: Kibana 7.4.0 Elastic 7.4.0 Logstash 7.4.0 Filebeat (33 instances) 6.8.3 We found out we were missing some log lines from two different filebeat clients at different times. After some searchi… --- ## [Logstash : Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release](https://discuss.elastic.co/t/logstash-java-hotspot-tm-64-bit-server-vm-warning-option-useconcmarksweepgc-was-deprecated-in-version-9-0-and-will-likely-be-removed-in-a-future-release/216060)
**Author:** [@Ekta](https://discuss.elastic.co/u/Ekta)\ **Replies:** 5\ **Last updated:** [January 23, 2020, 12:51pm UTC](https://discuss.elastic.co/t/logstash-java-hotspot-tm-64-bit-server-vm-warning-option-useconcmarksweepgc-was-deprecated-in-version-9-0-and-will-likely-be-removed-in-a-future-release/216060 "2020-01-23T12:51:53Z")
while shashing first log it show below warning: /usr/share/logstash# bin/logstash -e 'input { stdin { } } output { stdout {} }' Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in ver… --- ## [Logstash input plugin for Azure event Hub](https://discuss.elastic.co/t/logstash-input-plugin-for-azure-event-hub/216206)
**Author:** [@nujojo](https://discuss.elastic.co/u/nujojo)\ **Replies:** 0\ **Last updated:** [January 23, 2020, 9:41am UTC](https://discuss.elastic.co/t/logstash-input-plugin-for-azure-event-hub/216206 "2020-01-23T09:41:26Z")
Hi, I am looking into collecting VM logs from Azure event Hub with logstash. I was looking around, but was not able to find proper answer. I am looking plugin for Event Hub which is similar like AWS plugin: Do we … --- ## [Github suggest reviewer option](https://discuss.elastic.co/t/github-suggest-reviewer-option/216198)
**Author:** [@benoitdupont](https://discuss.elastic.co/u/benoitdupont)\ **Replies:** 2\ **Last updated:** [January 23, 2020, 9:14am UTC](https://discuss.elastic.co/t/github-suggest-reviewer-option/216198 "2020-01-23T09:14:47Z")
Hello, quick question: I proposed a PR on the logstash GitHub and I see that Github 'suggest' reviewers, with a request button. Is this the common way to do or the PR will be dealt with in a triage phase ? Cheers, Ben --- ## [Failed to query elasticsearch for previous event - no such index](https://discuss.elastic.co/t/failed-to-query-elasticsearch-for-previous-event-no-such-index/216185)
**Author:** [@mullan](https://discuss.elastic.co/u/mullan)\ **Replies:** 1\ **Last updated:** [January 23, 2020, 8:21am UTC](https://discuss.elastic.co/t/failed-to-query-elasticsearch-for-previous-event-no-such-index/216185 "2020-01-23T08:21:03Z")
Hi , My conf file look like this . input { jdbc { ..... statement =\> "Select userId as id,UserID as userid,firstName as firstname, lastName as lastname,countryCode as countrycode,countryName as countryname,bid as bi… --- ## [Change String to Hours - date filters](https://discuss.elastic.co/t/change-string-to-hours-date-filters/215834)
**Author:** [@katara](https://discuss.elastic.co/u/katara)\ **Replies:** 9\ **Last updated:** [January 23, 2020, 6:34am UTC](https://discuss.elastic.co/t/change-string-to-hours-date-filters/215834 "2020-01-23T06:34:18Z")
Hello everyone, I have a column like this: business\_time\_left 3 Hours 24 Minutes 59 Minutes 4 Days 23 Hours 58 Minutes 0 Seconds 1 Hour and so on.. What I want to do in Logstash is to convert this entirely into h… --- ## [Logstash Jdbc input Plugin configuration with SSL cert information](https://discuss.elastic.co/t/logstash-jdbc-input-plugin-configuration-with-ssl-cert-information/215639)
**Author:** [@mullan](https://discuss.elastic.co/u/mullan)\ **Replies:** 2\ **Last updated:** [January 23, 2020, 6:25am UTC](https://discuss.elastic.co/t/logstash-jdbc-input-plugin-configuration-with-ssl-cert-information/215639 "2020-01-23T06:25:28Z")
Hi, Iam new to Logstash. My project requires to configure logstash with jdbc input plugin.Below is the conf file I had used. input { jdbc { jdbc\_driver\_library =\> "lib\\jars\\mongodb\_unityjdbc\_full.jar" jdbc\_driver\_cl… --- ## [MIB OID Translation](https://discuss.elastic.co/t/mib-oid-translation/29710)
**Author:** [@tom\_jonge](https://discuss.elastic.co/u/tom_jonge)\ **Replies:** 32\ **Last updated:** [January 22, 2020, 3:13am UTC](https://discuss.elastic.co/t/mib-oid-translation/29710 "2020-01-22T03:13:37Z")
Dear logstash, /opt/logstash/vendor/bundle/jruby/1.9/gems/snmp-1.2.0/data/ruby/snmp/mibs containts a venerable treasure trove of pre-converted YAML MIBs. However I have run into an issue: From Kibana: RFC1155-SMI::… --- ## [Grok vs disect?](https://discuss.elastic.co/t/grok-vs-disect/216156)
**Author:** [@fallenreaper](https://discuss.elastic.co/u/fallenreaper)\ **Replies:** 0\ **Last updated:** [January 23, 2020, 12:03am UTC](https://discuss.elastic.co/t/grok-vs-disect/216156 "2020-01-23T00:03:06Z")
I was having a workshop today, and noticed that a guy kept bringing up dissect and that it is far more powerful from grok. To some extent. My question is, what is better for the use case? This is the filter in my lo… --- ## [Mixed JSON and other content throws a warning for JSON filter](https://discuss.elastic.co/t/mixed-json-and-other-content-throws-a-warning-for-json-filter/216152)
**Author:** [@Andrew\_LeCody](https://discuss.elastic.co/u/Andrew_LeCody)\ **Replies:** 0\ **Last updated:** [January 22, 2020, 11:04pm UTC](https://discuss.elastic.co/t/mixed-json-and-other-content-throws-a-warning-for-json-filter/216152 "2020-01-22T23:04:02Z")
Despite having skip\_on\_invalid\_json set to true I keep getting errors on content that isn't purely json, but is a mix of json and other content (null, int, strings). Here's my filter config: filter { json { sourc… --- ## [Logstash Error 413](https://discuss.elastic.co/t/logstash-error-413/216055)
**Author:** [@bhargav\_bharat](https://discuss.elastic.co/u/bhargav_bharat)\ **Replies:** 2\ **Last updated:** [January 22, 2020, 9:44pm UTC](https://discuss.elastic.co/t/logstash-error-413/216055 "2020-01-22T21:44:30Z")
Hello, I am facing below mentioned error and logstash stop sending log to elasticsearch and communication also broken. ########################################################### Jan 22 10:21:33 logstash\_agent logstas… --- ## [Elasticsearch unreachable error with logstash configuration](https://discuss.elastic.co/t/elasticsearch-unreachable-error-with-logstash-configuration/213043)
**Author:** [@Aniket\_Salvi](https://discuss.elastic.co/u/Aniket_Salvi)\ **Replies:** 4\ **Last updated:** [January 22, 2020, 9:12pm UTC](https://discuss.elastic.co/t/elasticsearch-unreachable-error-with-logstash-configuration/213043 "2020-01-22T21:12:23Z")
I am getting error when I am trying to access elasticsearch from logstash configuration. elasticsearch and logstash version - 7.4.0 logstash configuration :- input{ } output { elasticsearch { hosts =\> \["icici-elk-… --- ## [Can't get IIS Log row to match Grok-pattern](https://discuss.elastic.co/t/cant-get-iis-log-row-to-match-grok-pattern/215915)
**Author:** [@johanwallenborg](https://discuss.elastic.co/u/johanwallenborg)\ **Replies:** 4\ **Last updated:** [January 22, 2020, 5:03pm UTC](https://discuss.elastic.co/t/cant-get-iis-log-row-to-match-grok-pattern/215915 "2020-01-22T17:03:25Z")
Hello, I'm getting grokfailures on some of the lines on my IIS log parsing and currently i'm banging my head towards a wall. :slight\_smile: My pattern look like this: %{TIMESTAMP\_ISO8601:iis.access.time} %{IPORHOST:de… --- ## [Kafka input convert to json](https://discuss.elastic.co/t/kafka-input-convert-to-json/215785)
**Author:** [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Replies:** 5\ **Last updated:** [January 22, 2020, 4:23pm UTC](https://discuss.elastic.co/t/kafka-input-convert-to-json/215785 "2020-01-22T16:23:20Z")
I have use input as kafka and my message is like 2020-01-20 10:47:48: partition=0 offset=187965 machine=host1 queue=qhost1 comman=1298316293 action=REMOVE I want this to be separated out. how do I use filter input { … --- ## [Logstash can't seem to connect after years or working fine](https://discuss.elastic.co/t/logstash-cant-seem-to-connect-after-years-or-working-fine/216091)
**Author:** [@javadevmtl](https://discuss.elastic.co/u/javadevmtl)\ **Replies:** 0\ **Last updated:** [January 22, 2020, 3:59pm UTC](https://discuss.elastic.co/t/logstash-cant-seem-to-connect-after-years-or-working-fine/216091 "2020-01-22T15:59:30Z")
Hi, I have logstash 6.4.2 running inside docker container. The logstash instance was working fine for over a year. Just last few hours the below errors started hapening... I loged into the container itself and I pigned … --- ## [Query index by two fields](https://discuss.elastic.co/t/query-index-by-two-fields/216046)
**Author:** [@namali](https://discuss.elastic.co/u/namali)\ **Replies:** 5\ **Last updated:** [January 22, 2020, 3:45pm UTC](https://discuss.elastic.co/t/query-index-by-two-fields/216046 "2020-01-22T15:45:20Z")
how to find subempcategoryid from emp\_category index emp\_category index structure mainempcategoryid mainempcategoryname subempcategoryid subempcategoryname used elasticseaxh fillter pluging. but query not working … --- ## [How to read a text file using elastic search](https://discuss.elastic.co/t/how-to-read-a-text-file-using-elastic-search/216045)
**Author:** [@prudhvi](https://discuss.elastic.co/u/prudhvi)\ **Replies:** 6\ **Last updated:** [January 22, 2020, 2:50pm UTC](https://discuss.elastic.co/t/how-to-read-a-text-file-using-elastic-search/216045 "2020-01-22T14:50:01Z")
i am trying to read a text file using logstash and to push it to elasticsearch. How can i know if the input file is read, where can i see the data read by logstash. --- ## [\[logstash.outputs.elasticsearch\] Attempted to resurrect connection to dead ES instance](https://discuss.elastic.co/t/logstash-outputs-elasticsearch-attempted-to-resurrect-connection-to-dead-es-instance/216027)
**Author:** [@pyerunka](https://discuss.elastic.co/u/pyerunka)\ **Replies:** 3\ **Last updated:** [January 22, 2020, 2:05pm UTC](https://discuss.elastic.co/t/logstash-outputs-elasticsearch-attempted-to-resurrect-connection-to-dead-es-instance/216027 "2020-01-22T14:05:48Z")
Hello, I am trying to use jdbc input plugin using logstash so that i can fetch data from sql database and create the index. But when i am running config file for logstash. It is showing me below error: \[2020-01-22T10… --- ## [Remove alias winlog from fields](https://discuss.elastic.co/t/remove-alias-winlog-from-fields/216069)
**Author:** [@francescouk](https://discuss.elastic.co/u/francescouk)\ **Replies:** 0\ **Last updated:** [January 22, 2020, 1:28pm UTC](https://discuss.elastic.co/t/remove-alias-winlog-from-fields/216069 "2020-01-22T13:28:33Z")
Hi there, I would like to know if is possivel to remove winlog alias from winlogbeat fields? eg: winlog.channel =\> channel winlog.user.domain =\> user.domain ??? Thanks for the attention. --- ## [Remote tomcat logs](https://discuss.elastic.co/t/remote-tomcat-logs/216063)
**Author:** [@sagi](https://discuss.elastic.co/u/sagi)\ **Replies:** 0\ **Last updated:** [January 22, 2020, 1:01pm UTC](https://discuss.elastic.co/t/remote-tomcat-logs/216063 "2020-01-22T13:01:23Z")
I have configured ELK 6.8.2 on ubuntu 16.04, remote Syslog are loaded using filebeat , please anyone help me how to load remote tomcat logs are catalina.out. I am new to ELK. [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=414) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=416)