# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1069 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1070 --- ## [Scaling Logstash for Throttling](https://discuss.elastic.co/t/scaling-logstash-for-throttling/68005)
**Author:** [@nitin949](https://discuss.elastic.co/u/nitin949)\ **Replies:** 0\ **Last updated:** [December 5, 2016, 8:54am UTC](https://discuss.elastic.co/t/scaling-logstash-for-throttling/68005 "2016-12-05T08:54:05Z")
I was using Logstash for parsing logs. I am using Rsyslog + One Logstash instance + ES and Kibana. Now I am also doing throttling, that if particular types of request comes more than 10 per minute than generate an Ala… --- ## [Log4j logs and it import to logstash](https://discuss.elastic.co/t/log4j-logs-and-it-import-to-logstash/68003)
**Author:** [@Deepak\_Mann](https://discuss.elastic.co/u/Deepak_Mann)\ **Replies:** 0\ **Last updated:** [December 5, 2016, 8:42am UTC](https://discuss.elastic.co/t/log4j-logs-and-it-import-to-logstash/68003 "2016-12-05T08:42:50Z")
Hi, I had run a software which generates logs using log4j and saves them in a file ept.log the format is as follows launch TIMESTAMP START: 2016.11.15 05:30:23 Level: Message INFO: Batch lock retry wait = 30, max =… --- ## [Custom filters are not working with Logstash-5.0.0](https://discuss.elastic.co/t/custom-filters-are-not-working-with-logstash-5-0-0/65057)
**Author:** [@er.ashish1500](https://discuss.elastic.co/u/er.ashish1500)\ **Replies:** 7\ **Last updated:** [December 5, 2016, 8:23am UTC](https://discuss.elastic.co/t/custom-filters-are-not-working-with-logstash-5-0-0/65057 "2016-12-05T08:23:20Z")
I am new for logstash 5.0 and tried to create custom filter at 'PATH/logstash/filters/global\_config.rb' with the name of global config but I am getting error as: bin/logstash -f conf\_d/ Sending Logstash logs to /usr/loc… --- ## [Parser question](https://discuss.elastic.co/t/parser-question/66843)
**Author:** [@bharath\_meka](https://discuss.elastic.co/u/bharath_meka)\ **Replies:** 2\ **Last updated:** [December 5, 2016, 8:09am UTC](https://discuss.elastic.co/t/parser-question/66843 "2016-12-05T08:09:29Z")
Hi, Any clue why the below parser is failing? Parser SPARK\_TIMESTAMP %{DATE\_EU} %{TIME} %{SPARK\_TIMESTAMP:timestamp} %{LOGLEVEL:loglevel} %{WORD:classname}(:)? (\[%{DATA}\]\[UN-%{DATA:userid}\]\[(HSI|ISI)-%{DATA:sessionid… --- ## [How to parse logs](https://discuss.elastic.co/t/how-to-parse-logs/67705)
**Author:** [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\ **Replies:** 3\ **Last updated:** [December 5, 2016, 6:58am UTC](https://discuss.elastic.co/t/how-to-parse-logs/67705 "2016-12-05T06:58:43Z")
Hello All, I have a amazon redshift logging been stored on amazon s3. I want to use the data from this s3 bucket to visualize. Below are some sample logs which is generated from redshift. '2016-11-29T00:00:04Z UTC \[ … --- ## [Losgstash output to influxdb](https://discuss.elastic.co/t/losgstash-output-to-influxdb/67495)
**Author:** [@sparashara](https://discuss.elastic.co/u/sparashara)\ **Replies:** 3\ **Last updated:** [December 5, 2016, 6:55am UTC](https://discuss.elastic.co/t/losgstash-output-to-influxdb/67495 "2016-12-05T06:55:57Z")
HI All, kindly help me on below time series data { "data\_points": "{\\"HOST-4B1811DD3A99AEEE\\":\[\[1480407000000,7.572072347005208\],\[1480407300000,10.174551391601563\],\[1480407600000,11.274533081054688\],\[1480407900000,10… --- ## [Logstash - template/index issue](https://discuss.elastic.co/t/logstash-template-index-issue/67669)
**Author:** [@Gude.ravi](https://discuss.elastic.co/u/Gude.ravi)\ **Replies:** 1\ **Last updated:** [December 5, 2016, 6:43am UTC](https://discuss.elastic.co/t/logstash-template-index-issue/67669 "2016-12-05T06:43:27Z")
Before loading my data, I checked that my index (securedapachelogs) or template ( sc\_apache\_template) in elasticsearch. both do not exist. Logstash is going to request that template/index were created on startup. Once th… --- ## [Index a document from Logstash using the ‘document\_id](https://discuss.elastic.co/t/index-a-document-from-logstash-using-the-document-id/67651)
**Author:** [@P-F](https://discuss.elastic.co/u/P-F)\ **Replies:** 1\ **Last updated:** [December 5, 2016, 6:40am UTC](https://discuss.elastic.co/t/index-a-document-from-logstash-using-the-document-id/67651 "2016-12-05T06:40:02Z")
Hi Basically we want to be able to index a document from Logstash using the ‘document\_id’ parameter in the Elasticsearch output to overwrite\\update a document and also use curator to delete anything older than 30days. W… --- ## [Logstash output to self, can't receive anything?](https://discuss.elastic.co/t/logstash-output-to-self-cant-receive-anything/67639)
**Author:** [@skluoyifan](https://discuss.elastic.co/u/skluoyifan)\ **Replies:** 1\ **Last updated:** [December 5, 2016, 6:35am UTC](https://discuss.elastic.co/t/logstash-output-to-self-cant-receive-anything/67639 "2016-12-05T06:35:11Z")
config like this: input { file { path =\> "D:/Projects/jzt2/bd/erpxs/logstash/example.json" codec =\> json { charset =\> "UTF-8" } start\_position =\> beginning sincedb\_path =\> "./sincedb.txt" typ… --- ## [RuntimeError: LogStash::Filters::Translate: can't convert Array into Hash when loading dictionary](https://discuss.elastic.co/t/runtimeerror-logstash-filters-translate-cant-convert-array-into-hash-when-loading-dictionary/67632)
**Author:** [@imperfection\_perfect](https://discuss.elastic.co/u/imperfection_perfect)\ **Replies:** 1\ **Last updated:** [December 5, 2016, 6:34am UTC](https://discuss.elastic.co/t/runtimeerror-logstash-filters-translate-cant-convert-array-into-hash-when-loading-dictionary/67632 "2016-12-05T06:34:03Z")
Hi Team, I have the following usescase, I am on logstash 2.x installed the logstash translate filter logstash-filter-translate-2.1.4 3letterISOCountryCodeToLatLongMapping but this is not working and i get a r… --- ## [Windows Snare to Logstash](https://discuss.elastic.co/t/windows-snare-to-logstash/67628)
**Author:** [@Zeus](https://discuss.elastic.co/u/Zeus)\ **Replies:** 1\ **Last updated:** [December 5, 2016, 6:31am UTC](https://discuss.elastic.co/t/windows-snare-to-logstash/67628 "2016-12-05T06:31:08Z")
I am brand new to this software. I have been searching for information on how to correctly get Logstash to see logs from Snare with no success. I currently have about 20 servers half Linux half windows. All Linux machine… --- ## [SSL problem with logstash newrelic output plugin](https://discuss.elastic.co/t/ssl-problem-with-logstash-newrelic-output-plugin/67979)
**Author:** [@charekc](https://discuss.elastic.co/u/charekc)\ **Replies:** 0\ **Last updated:** [December 5, 2016, 4:44am UTC](https://discuss.elastic.co/t/ssl-problem-with-logstash-newrelic-output-plugin/67979 "2016-12-05T04:44:37Z")
Hello I am running logstash 5.0.2 and using output plugin new relic 0.9.x My output config is like the following newrelic { account\_id =\> "MyAccountId1111111111" insert\_key =\> "myInsertKeyXYZ" event\_type =\> "Myevent" } … --- ## [Metricbeat -\> LS -\> ES, template error \[SOLVED\]](https://discuss.elastic.co/t/metricbeat-ls-es-template-error-solved/64671)
**Author:** [@gregoryo](https://discuss.elastic.co/u/gregoryo)\ **Replies:** 3\ **Last updated:** [December 5, 2016, 4:41am UTC](https://discuss.elastic.co/t/metricbeat-ls-es-template-error-solved/64671 "2016-12-05T04:41:02Z")
I have been successfully using Filebeat -\> LS -\> ES for a few months. I now want to add Metricbeat as another source. I started ingesting data fine, but continued reading docs. https://www.elastic.co/guide/en/beats/metri… --- ## [How can i fix this error for date type](https://discuss.elastic.co/t/how-can-i-fix-this-error-for-date-type/67969)
**Author:** [@saisn](https://discuss.elastic.co/u/saisn)\ **Replies:** 0\ **Last updated:** [December 5, 2016, 1:29am UTC](https://discuss.elastic.co/t/how-can-i-fix-this-error-for-date-type/67969 "2016-12-05T01:29:50Z")
Hi , I'm getting below error when trying to insert records from SQL using logstash "status"=\>400, "error"=\>{"type"=\>"mapper\_parsing\_exception", "reason"=\>"failed to parse", "caused\_by"=\>{"type"=\>"illegal\_argument\_ex… --- ## [Multiple files from filebeat](https://discuss.elastic.co/t/multiple-files-from-filebeat/67965)
**Author:** [@bob-bza](https://discuss.elastic.co/u/bob-bza)\ **Replies:** 1\ **Last updated:** [December 4, 2016, 11:33pm UTC](https://discuss.elastic.co/t/multiple-files-from-filebeat/67965 "2016-12-04T23:33:12Z")
i have multiple files i was trying to index all those with different index's , i was wondering how can i manage those. filebeat i was forwarding to logstash and then output to elastic search. Example: /var/log/file1 /v… --- ## [Jdbc error : NoMethodError: undefined method \`message=' for #\ **Author:** [@gilisade](https://discuss.elastic.co/u/gilisade)\ **Replies:** 1\ **Last updated:** [December 4, 2016, 11:22am UTC](https://discuss.elastic.co/t/jdbc-error-nomethoderror-undefined-method-message-for-java/67606 "2016-12-04T11:22:04Z") Hi there running Logstash 5.0.1 ruby version: 2.0.0p598 (2014-11-13) \[x86\_64-linux\] trying to extract data from a MSSQL server with the following config: jdbc { jdbc\_driver\_library =\> /path/to/sqljd… --- ## [How to config redis-sentinel for input and output?](https://discuss.elastic.co/t/how-to-config-redis-sentinel-for-input-and-output/67940)
**Author:** [@kaybinwong](https://discuss.elastic.co/u/kaybinwong)\ **Replies:** 0\ **Last updated:** [December 4, 2016, 5:00am UTC](https://discuss.elastic.co/t/how-to-config-redis-sentinel-for-input-and-output/67940 "2016-12-04T05:00:04Z")
how to config redis-sentinel for input and output? --- ## [SearchContextMissingException - Duplication at Output](https://discuss.elastic.co/t/searchcontextmissingexception-duplication-at-output/67914)
**Author:** [@gringo](https://discuss.elastic.co/u/gringo)\ **Replies:** 0\ **Last updated:** [December 3, 2016, 1:19am UTC](https://discuss.elastic.co/t/searchcontextmissingexception-duplication-at-output/67914 "2016-12-03T01:19:10Z")
Anyone knows what is this? duplicate org.elasticsearch.search.SearchContextMissingException: No search context found for id I keep getting this in my logstash when I opened up many indexes on the elasticsearch server… --- ## [Logstash Freezes with Kafka Output Plugin](https://discuss.elastic.co/t/logstash-freezes-with-kafka-output-plugin/67906)
**Author:** [@raghavsys](https://discuss.elastic.co/u/raghavsys)\ **Replies:** 0\ **Last updated:** [December 2, 2016, 10:26pm UTC](https://discuss.elastic.co/t/logstash-freezes-with-kafka-output-plugin/67906 "2016-12-02T22:26:02Z")
Logstash version: 2.3.3 Kafka version: 0.8.2 We have been using logstash to parse the input files and apply few basic filters (adding fields). The output plugin is Kafka. Most of the time, the logstash freezes and do… --- ## [Logstash 5.0.2 not removing tags field](https://discuss.elastic.co/t/logstash-5-0-2-not-removing-tags-field/67604)
**Author:** [@xerxes9000sx](https://discuss.elastic.co/u/xerxes9000sx)\ **Replies:** 4\ **Last updated:** [December 2, 2016, 8:29pm UTC](https://discuss.elastic.co/t/logstash-5-0-2-not-removing-tags-field/67604 "2016-12-02T20:29:38Z")
I remove tags wield via mutate remove\_fields statement. Logstash 5.0.1 and 5.0.2 is not removing tags field. Logstash 5.0.0 and 2.4.1 is removing tags field. Logstash test config: input { stdin { tags =\> \["test"\] … --- ## [Is there a way to map a date to milliseconds since epoch for Logstash?](https://discuss.elastic.co/t/is-there-a-way-to-map-a-date-to-milliseconds-since-epoch-for-logstash/67798)
**Author:** [@EuclideanSearch](https://discuss.elastic.co/u/EuclideanSearch)\ **Replies:** 5\ **Last updated:** [December 2, 2016, 8:25pm UTC](https://discuss.elastic.co/t/is-there-a-way-to-map-a-date-to-milliseconds-since-epoch-for-logstash/67798 "2016-12-02T20:25:47Z")
Hi, I have a date field (timestmap) (ES type is date) in my current Elasticsearch cluster and, among other things, I want to use Logstash to create a field consisting of the milliseconds since epoch for the timestamp. … --- ## [Salesforce Input - Fetch Frequency](https://discuss.elastic.co/t/salesforce-input-fetch-frequency/44354)
**Author:** [@bhatch](https://discuss.elastic.co/u/bhatch)\ **Replies:** 4\ **Last updated:** [December 2, 2016, 5:57pm UTC](https://discuss.elastic.co/t/salesforce-input-fetch-frequency/44354 "2016-12-02T17:57:43Z")
We are looking into ways to get some data from Salesforce into Elasticsearch. Mainly to help correlate customer support and sales events with our various log data. However I am having trouble determining how often the I… --- ## [Convert fieldtype from neflow protocol .uint8](https://discuss.elastic.co/t/convert-fieldtype-from-neflow-protocol-uint8/67873)
**Author:** [@pereyrdi](https://discuss.elastic.co/u/pereyrdi)\ **Replies:** 1\ **Last updated:** [December 2, 2016, 3:59pm UTC](https://discuss.elastic.co/t/convert-fieldtype-from-neflow-protocol-uint8/67873 "2016-12-02T15:59:08Z")
Hi all ! Im creating new fields from netflow information, for example get the ipaddres and resolve the hostname into another field. mutate { add\_field =\> { "IPAddress" =\> "%{host}" } add\_field =\> { "Hos… --- ## [Logstash 2.1.1 - weird results with multiple conf files on WIndows](https://discuss.elastic.co/t/logstash-2-1-1-weird-results-with-multiple-conf-files-on-windows/42043)
**Author:** [@kpdude77](https://discuss.elastic.co/u/kpdude77)\ **Replies:** 4\ **Last updated:** [December 2, 2016, 12:08pm UTC](https://discuss.elastic.co/t/logstash-2-1-1-weird-results-with-multiple-conf-files-on-windows/42043 "2016-12-02T12:08:04Z")
I am running logstash 2.1.1 on a local WIndows system with 2 conf files - one with a jdbc input that defines a custom index in the elasticsearch output, and one with a file input from a log that uses the default index (… --- ## [Scheduling a logsatsh config file](https://discuss.elastic.co/t/scheduling-a-logsatsh-config-file/67831)
**Author:** [@sabyasachi\_mallick](https://discuss.elastic.co/u/sabyasachi_mallick)\ **Replies:** 7\ **Last updated:** [December 2, 2016, 11:05am UTC](https://discuss.elastic.co/t/scheduling-a-logsatsh-config-file/67831 "2016-12-02T11:05:42Z")
i have 4 independent logstash config files which are taking input from a csv file .i am executing 4 files in background by using logstash -f /path/to/config file. But when my csv file is updated with new data,then all t… --- ## [How can I set elasticsearch type in logstash conf file when importing?](https://discuss.elastic.co/t/how-can-i-set-elasticsearch-type-in-logstash-conf-file-when-importing/67579)
**Author:** [@tanimoto](https://discuss.elastic.co/u/tanimoto)\ **Replies:** 12\ **Last updated:** [December 2, 2016, 9:00am UTC](https://discuss.elastic.co/t/how-can-i-set-elasticsearch-type-in-logstash-conf-file-when-importing/67579 "2016-12-02T09:00:19Z")
I'm trying to import CSV file to elasticsearch through logstash. data can be imported and type name is logs. Is there any way to change type name? Any setting to type name in logstash conf file when importing?? --- ## [Elasticsearch-Output to Elasticsearch-Cluster](https://discuss.elastic.co/t/elasticsearch-output-to-elasticsearch-cluster/65532)
**Author:** [@TWalter](https://discuss.elastic.co/u/TWalter)\ **Replies:** 4\ **Last updated:** [December 2, 2016, 8:15am UTC](https://discuss.elastic.co/t/elasticsearch-output-to-elasticsearch-cluster/65532 "2016-12-02T08:15:14Z")
Hello there, i have an Elasticsearch-Cluster with three nodes. When i import the data to the cluster with the Elasticsearch-Output-Plugin, which node do i choose as Destination? Is there a common expression for the … --- ## [LS1.5 output to file, dots](https://discuss.elastic.co/t/ls1-5-output-to-file-dots/67808)
**Author:** [@teebu](https://discuss.elastic.co/u/teebu)\ **Replies:** 1\ **Last updated:** [December 2, 2016, 7:23am UTC](https://discuss.elastic.co/t/ls1-5-output-to-file-dots/67808 "2016-12-02T07:23:55Z")
LS1.5 I want to output docs to a file, but file{} is outputting the whole event. i tried using: file {codec =\> dots} and file {codec =\> line {format =\> "."}} but its not doing it. It outputs the entire event json. --- ## [Rabbitmq output plugin, how to set message headers?](https://discuss.elastic.co/t/rabbitmq-output-plugin-how-to-set-message-headers/67811)
**Author:** [@TonyEnnis](https://discuss.elastic.co/u/TonyEnnis)\ **Replies:** 1\ **Last updated:** [December 2, 2016, 7:05am UTC](https://discuss.elastic.co/t/rabbitmq-output-plugin-how-to-set-message-headers/67811 "2016-12-02T07:05:10Z")
Just what the subject says. I need to be set a value in an outbound rabbitmq message's header. There is evidence that this has been implemented, but I see nothing in the documentation. --- ## [How to schedule jdbc to run on every sunday?](https://discuss.elastic.co/t/how-to-schedule-jdbc-to-run-on-every-sunday/67338)
**Author:** [@sabyasachi\_mallick](https://discuss.elastic.co/u/sabyasachi_mallick)\ **Replies:** 2\ **Last updated:** [December 2, 2016, 6:24am UTC](https://discuss.elastic.co/t/how-to-schedule-jdbc-to-run-on-every-sunday/67338 "2016-12-02T06:24:35Z")
i want to update my elastic data source on weekly basis.And i want it to do on every sunday rather than updating in every 7 days.Is there any way to do this? [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1068) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1070)