# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1102 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1103 --- ## [Logstash is not reading CSV file](https://discuss.elastic.co/t/logstash-is-not-reading-csv-file/60569)
**Author:** [@Senz79](https://discuss.elastic.co/u/Senz79)\ **Replies:** 3\ **Last updated:** [September 15, 2016, 8:13am UTC](https://discuss.elastic.co/t/logstash-is-not-reading-csv-file/60569 "2016-09-15T08:13:41Z")
i am trying to read a CSV file and seems Logstash is not reading the file. Currently it is stuck at D:\\Tools\\ELK\\logstash-2.4.0\\bin\>logstash --allow-env agent -f test.conf Settings: Default pipeline workers: 4 Pipelin… --- ## [\[RESOLVED\] Is there a way to store the whole input as a subfield?](https://discuss.elastic.co/t/resolved-is-there-a-way-to-store-the-whole-input-as-a-subfield/60509)
**Author:** [@vandamo](https://discuss.elastic.co/u/vandamo)\ **Replies:** 4\ **Last updated:** [September 15, 2016, 7:36am UTC](https://discuss.elastic.co/t/resolved-is-there-a-way-to-store-the-whole-input-as-a-subfield/60509 "2016-09-15T07:36:06Z")
Hi, I've been asked to save all events in Elasticsearch in 2 separate indexes: - one that is "classic" with a dynamic mapping - one that is "raw": the whole incoming event should be stored as a string, without any attemp… --- ## [Not able to index the logs coming from Shipper through redis into the ELK](https://discuss.elastic.co/t/not-able-to-index-the-logs-coming-from-shipper-through-redis-into-the-elk/60367)
**Author:** [@ANU\_SARA\_VARGHESE](https://discuss.elastic.co/u/ANU_SARA_VARGHESE)\ **Replies:** 11\ **Last updated:** [September 15, 2016, 7:19am UTC](https://discuss.elastic.co/t/not-able-to-index-the-logs-coming-from-shipper-through-redis-into-the-elk/60367 "2016-09-15T07:19:38Z")
Hi, I am deploying ELK for POC. In that, I have pushed the logs into the logstash shipper(I am able to see that using tcpdump). But then I am not able to see them in Elasticsearch and kibana. I am able to see them at R… --- ## [Unique ID](https://discuss.elastic.co/t/unique-id/60568)
**Author:** [@gringo](https://discuss.elastic.co/u/gringo)\ **Replies:** 3\ **Last updated:** [September 15, 2016, 6:57am UTC](https://discuss.elastic.co/t/unique-id/60568 "2016-09-15T06:57:46Z")
How to create a unique id for all events using logstash? This is something like incremental unique ID used in a normal database. --- ## [How to get the ouput](https://discuss.elastic.co/t/how-to-get-the-ouput/60389)
**Author:** [@sweta](https://discuss.elastic.co/u/sweta)\ **Replies:** 4\ **Last updated:** [September 15, 2016, 5:49am UTC](https://discuss.elastic.co/t/how-to-get-the-ouput/60389 "2016-09-15T05:49:22Z")
can any one tell me why i am not getting the output.I am very new in this technology and started learning it from last week i dont know what is the problem and how i can solve this problem. --- ## [Logstash UDP Message Missing Beginning](https://discuss.elastic.co/t/logstash-udp-message-missing-beginning/58858)
**Author:** [@r0b0tAnthony](https://discuss.elastic.co/u/r0b0tAnthony)\ **Replies:** 3\ **Last updated:** [September 14, 2016, 8:27pm UTC](https://discuss.elastic.co/t/logstash-udp-message-missing-beginning/58858 "2016-09-14T20:27:20Z")
Hi, I am forwarding a syslog from a device to Logstash using port 5514. However, my syslog grok,which works great for any filebeat shipped syslogs, fails for the UDP forwarded syslogs. I have tested the grok using the … --- ## [Plugin install via package manager](https://discuss.elastic.co/t/plugin-install-via-package-manager/60542)
**Author:** [@samcmill](https://discuss.elastic.co/u/samcmill)\ **Replies:** 0\ **Last updated:** [September 14, 2016, 8:02pm UTC](https://discuss.elastic.co/t/plugin-install-via-package-manager/60542 "2016-09-14T20:02:12Z")
Hi, I have the requirement that everything must be installed via the Linux distribution package manager. I need to install a couple of Logstash plugins. I'm looking for guidance on how to best install the plugins via… --- ## [Big-query Plugin Date suffix Improvement](https://discuss.elastic.co/t/big-query-plugin-date-suffix-improvement/60535)
**Author:** [@Ngumba\_Kamau](https://discuss.elastic.co/u/Ngumba_Kamau)\ **Replies:** 0\ **Last updated:** [September 14, 2016, 5:48pm UTC](https://discuss.elastic.co/t/big-query-plugin-date-suffix-improvement/60535 "2016-09-14T17:48:08Z")
Currently the big-query plugin allows the user to select the prefix and then affixes the current date to the end of the table, for our projects we have come across the issue in which we would like to use the time stamp o… --- ## [Parsing date field value using grok match](https://discuss.elastic.co/t/parsing-date-field-value-using-grok-match/60506)
**Author:** [@rajkamalkool6](https://discuss.elastic.co/u/rajkamalkool6)\ **Replies:** 1\ **Last updated:** [September 14, 2016, 4:42pm UTC](https://discuss.elastic.co/t/parsing-date-field-value-using-grok-match/60506 "2016-09-14T16:42:46Z")
Hi, I am trying to parse the date field timestamp using grok expression under grok match plugin. The fields which I have given under grok match didn't generated. grok{ match =\> {"timestamp" =\> "%{MONTH:month} %{MON… --- ## [Adding/configuring available visualization fields](https://discuss.elastic.co/t/adding-configuring-available-visualization-fields/59467)
**Author:** [@hburnswell](https://discuss.elastic.co/u/hburnswell)\ **Replies:** 18\ **Last updated:** [September 14, 2016, 4:36pm UTC](https://discuss.elastic.co/t/adding-configuring-available-visualization-fields/59467 "2016-09-14T16:36:23Z")
All, I am new to ELK and have been reading and watching docs and videos for the last week. I have everything (Elasticsearch, Logstash, Kibana, filebeat) installed and have a dev apache server sending access and error l… --- ## [How to deal with "Previous message repeated 2 times"](https://discuss.elastic.co/t/how-to-deal-with-previous-message-repeated-2-times/60516)
**Author:** [@Mick\_Mahoney](https://discuss.elastic.co/u/Mick_Mahoney)\ **Replies:** 0\ **Last updated:** [September 14, 2016, 2:43pm UTC](https://discuss.elastic.co/t/how-to-deal-with-previous-message-repeated-2-times/60516 "2016-09-14T14:43:26Z")
Hi, Wonder if anyone can advise how to deal with processing an erro log file which can be broken down easily into fields, however there is the occasional line such as: "Previous message repeated X times", where X is … --- ## [Logstash jdbc input filter broken](https://discuss.elastic.co/t/logstash-jdbc-input-filter-broken/60507)
**Author:** [@biswajit86](https://discuss.elastic.co/u/biswajit86)\ **Replies:** 0\ **Last updated:** [September 14, 2016, 1:39pm UTC](https://discuss.elastic.co/t/logstash-jdbc-input-filter-broken/60507 "2016-09-14T13:39:13Z")
Hi, I am using the tracking\_column functionality of logstash jdbc input plugin to fetch database values from the db. Right now, the functionality to write the last :sql\_last\_value" into the metadata path seems to brok… --- ## [How to parse different types log files in logstash?](https://discuss.elastic.co/t/how-to-parse-different-types-log-files-in-logstash/60462)
**Author:** [@dvkawa](https://discuss.elastic.co/u/dvkawa)\ **Replies:** 7\ **Last updated:** [September 14, 2016, 12:44pm UTC](https://discuss.elastic.co/t/how-to-parse-different-types-log-files-in-logstash/60462 "2016-09-14T12:44:36Z")
Hi, I have two different weblogic log files, my conf file looks like below but it doesn't work: input { file { path =\> "D:/\/logs/ecomm\_vws02\_app" … --- ## [Maintain state between polling intervals](https://discuss.elastic.co/t/maintain-state-between-polling-intervals/60420)
**Author:** [@jbarak](https://discuss.elastic.co/u/jbarak)\ **Replies:** 2\ **Last updated:** [September 14, 2016, 12:07pm UTC](https://discuss.elastic.co/t/maintain-state-between-polling-intervals/60420 "2016-09-14T12:07:25Z")
I am using the http\_poller input plugin to poll a rest api every n secs. I need to store a value from the response of the current interval and use it in the next interval. Is there anyway to store/maintain this informa… --- ## [Field missing in Elasticsearch/kibana](https://discuss.elastic.co/t/field-missing-in-elasticsearch-kibana/60381)
**Author:** [@trenzalore](https://discuss.elastic.co/u/trenzalore)\ **Replies:** 4\ **Last updated:** [September 14, 2016, 7:25am UTC](https://discuss.elastic.co/t/field-missing-in-elasticsearch-kibana/60381 "2016-09-14T07:25:12Z")
Hi ! So I made a huge logstash config for my wifi hotspots. I managed to create nearly all the filters but I have a problem with one of them. I receive this log: \<134\>1 1473769601.251850936 APVDA001 flows allow src… --- ## [Referencing an external file](https://discuss.elastic.co/t/referencing-an-external-file/60496)
**Author:** [@debarun](https://discuss.elastic.co/u/debarun)\ **Replies:** 1\ **Last updated:** [September 14, 2016, 11:47am UTC](https://discuss.elastic.co/t/referencing-an-external-file/60496 "2016-09-14T11:47:45Z")
HI, I have a scenario where I want to match a log entry with multiple IP addresses and website names. I want to check if any of these IPs and addresses are in the forbidden list, To do that i think i need to check if… --- ## [Sleep condition is not working in logstash](https://discuss.elastic.co/t/sleep-condition-is-not-working-in-logstash/60494)
**Author:** [@Ganesh2303](https://discuss.elastic.co/u/Ganesh2303)\ **Replies:** 0\ **Last updated:** [September 14, 2016, 11:33am UTC](https://discuss.elastic.co/t/sleep-condition-is-not-working-in-logstash/60494 "2016-09-14T11:33:46Z")
HI Team, I want to process request and response should delay the time between this two files. So i have implemented the sleep concept. In early it was work for me and now try to implement that in logstash it through som… --- ## [How to covert boolean to integer](https://discuss.elastic.co/t/how-to-covert-boolean-to-integer/60483)
**Author:** [@Chita\_Rain](https://discuss.elastic.co/u/Chita_Rain)\ **Replies:** 1\ **Last updated:** [September 14, 2016, 10:39am UTC](https://discuss.elastic.co/t/how-to-covert-boolean-to-integer/60483 "2016-09-14T10:39:25Z")
try to use mutate { conver =\> \["type", 'integer'\] } but I got error NoMethodError: undefined method \`to\_i' for false:FalseClass convert\_integer at /opt/logstash/vendor/bundle… --- ## [Grok result can not find my data field](https://discuss.elastic.co/t/grok-result-can-not-find-my-data-field/60458)
**Author:** [@Chita\_Rain](https://discuss.elastic.co/u/Chita_Rain)\ **Replies:** 2\ **Last updated:** [September 14, 2016, 9:18am UTC](https://discuss.elastic.co/t/grok-result-can-not-find-my-data-field/60458 "2016-09-14T09:18:49Z")
I try to use Grok to match some log like below: \[2016-09-07 00:00:02\] online.INFO: GenerateLogData: {"userType":"MERCHANT","targetPlatformUserNo":"5mwe"} with pattern : \\\[%{TIMESTAMP\_ISO8601:timestamp}\\\] %{DATA:env}\\… --- ## [Logstash webhdfs example with Kerberos enabled hadoop cluster](https://discuss.elastic.co/t/logstash-webhdfs-example-with-kerberos-enabled-hadoop-cluster/51941)
**Author:** [@wuka1](https://discuss.elastic.co/u/wuka1)\ **Replies:** 1\ **Last updated:** [September 14, 2016, 7:50am UTC](https://discuss.elastic.co/t/logstash-webhdfs-example-with-kerberos-enabled-hadoop-cluster/51941 "2016-09-14T07:50:44Z")
configured the webhdfs\_helper.rb that client.kerberos = true /client.kerberos\_keytab="xxx" and installed the gssapi but still have such problem: Webhdfs check request failed. (namenode: 10.0.0.37:14000, Exception: undef… --- ## [Logstash JDBC Statement with Unicode](https://discuss.elastic.co/t/logstash-jdbc-statement-with-unicode/59412)
**Author:** [@yaacovki](https://discuss.elastic.co/u/yaacovki)\ **Replies:** 5\ **Last updated:** [September 14, 2016, 5:34am UTC](https://discuss.elastic.co/t/logstash-jdbc-statement-with-unicode/59412 "2016-09-14T05:34:52Z")
Hi, I'm trying to Run an Oracle SQL Statement with JDBC. the SQL Statement has some unicode text in it (Hebrew) in order to filter information. Logstash cannot run this. this is a basic example that isnt working n… --- ## [Filter to be used for atlassian products](https://discuss.elastic.co/t/filter-to-be-used-for-atlassian-products/60392)
**Author:** [@chagara](https://discuss.elastic.co/u/chagara)\ **Replies:** 2\ **Last updated:** [September 14, 2016, 12:42am UTC](https://discuss.elastic.co/t/filter-to-be-used-for-atlassian-products/60392 "2016-09-14T00:42:10Z")
Helo, We are going to use filebeat to send the logs from atlassian products(jira,confluence etc) to our ELK environment but, I have not been able to figure out how to set up a filter for those logs. Does anyone have an… --- ## [Where to get logstash kafka output plugin 4.x](https://discuss.elastic.co/t/where-to-get-logstash-kafka-output-plugin-4-x/60442)
**Author:** [@Da\_Zhou](https://discuss.elastic.co/u/Da_Zhou)\ **Replies:** 0\ **Last updated:** [September 14, 2016, 12:18am UTC](https://discuss.elastic.co/t/where-to-get-logstash-kafka-output-plugin-4-x/60442 "2016-09-14T00:18:09Z")
Hi , I need to work on logstash kafka output plugin, and I need its version to be 4.xx ot 3.xxx. I tried to checkout its branch on git hub, but it only have master branch(5.X) and a 2.x branch. So where can I find t… --- ## [Gzip input plugin not working for local gzip files?](https://discuss.elastic.co/t/gzip-input-plugin-not-working-for-local-gzip-files/60440)
**Author:** [@Allie\_Yang](https://discuss.elastic.co/u/Allie_Yang)\ **Replies:** 0\ **Last updated:** [September 13, 2016, 10:29pm UTC](https://discuss.elastic.co/t/gzip-input-plugin-not-working-for-local-gzip-files/60440 "2016-09-13T22:29:41Z")
i am wondering if logstash can read local gzip files? It never worked before. Below is my config. Appreciate some advice, tks! input { file { path =\> "/Users/Desktop/log\_file/test.log.gz" start\_position =\> "b… --- ## [Copy a field without making it a hash value?](https://discuss.elastic.co/t/copy-a-field-without-making-it-a-hash-value/60342)
**Author:** [@Allie\_Yang](https://discuss.elastic.co/u/Allie_Yang)\ **Replies:** 6\ **Last updated:** [September 13, 2016, 7:51pm UTC](https://discuss.elastic.co/t/copy-a-field-without-making-it-a-hash-value/60342 "2016-09-13T19:51:13Z")
Hi I have an array field. And i want to copy it without turning it to a hash data type, which is what "add\_field" does. Is there any ideas? Tks! PS: the reason i want to do it is because i now have field search\_terms =… --- ## [Logstash automatic configuration reload CPU consumption](https://discuss.elastic.co/t/logstash-automatic-configuration-reload-cpu-consumption/60415)
**Author:** [@Steve\_McDuff](https://discuss.elastic.co/u/Steve_McDuff)\ **Replies:** 0\ **Last updated:** [September 13, 2016, 5:46pm UTC](https://discuss.elastic.co/t/logstash-automatic-configuration-reload-cpu-consumption/60415 "2016-09-13T17:46:06Z")
I'm currently running Logstash with automated configuration reload so that I may automate the deployment of new configuration without triggering a restart. I noticed that automated configuration reload is quite CPU int… --- ## [{Help me}Extracting Domain from Mail address](https://discuss.elastic.co/t/help-me-extracting-domain-from-mail-address/60186)
**Author:** [@Uriel\_Ricardo](https://discuss.elastic.co/u/Uriel_Ricardo)\ **Replies:** 6\ **Last updated:** [September 13, 2016, 5:45pm UTC](https://discuss.elastic.co/t/help-me-extracting-domain-from-mail-address/60186 "2016-09-13T17:45:46Z")
Hello Friends. I need to extract the domain of an email address , I have a .csv , in one of the columns it stores the e -mail. How can I do it only get the domain? Note : This is my csv filter, it stores the e- mail … --- ## [Geo\_point](https://discuss.elastic.co/t/geo-point/48215)
**Author:** [@echua](https://discuss.elastic.co/u/echua)\ **Replies:** 5\ **Last updated:** [September 13, 2016, 4:39pm UTC](https://discuss.elastic.co/t/geo-point/48215 "2016-09-13T16:39:31Z")
I am trying to use the mapping portion of Kibana to map all the points that I have in my database for a given time period. I just started ELK so I don't know a lot. I have been struggling to insert geo\_point type of da… --- ## [Logstash Alpha 5 - Translation Missing error](https://discuss.elastic.co/t/logstash-alpha-5-translation-missing-error/60391)
**Author:** [@khall](https://discuss.elastic.co/u/khall)\ **Replies:** 1\ **Last updated:** [September 13, 2016, 3:41pm UTC](https://discuss.elastic.co/t/logstash-alpha-5-translation-missing-error/60391 "2016-09-13T15:41:21Z")
Can someone please direct me in how to fix this error I'm getting? I'm trying to pull in some sample data to test Alpha5 on an MS Windows platform: C:\\ELK\\ESTACK5\\LS5\\LSALPHA5\\bin\>logstash.bat -f c:\\ELK\\Datasets\\DAT… --- ## [Logstash worker parameter](https://discuss.elastic.co/t/logstash-worker-parameter/60291)
**Author:** [@il.bert](https://discuss.elastic.co/u/il.bert)\ **Replies:** 1\ **Last updated:** [September 13, 2016, 3:08pm UTC](https://discuss.elastic.co/t/logstash-worker-parameter/60291 "2016-09-13T15:08:27Z")
Hi everybody, I am trying to tune my ELK stack, but I don't get the meaning of some parameters in logstash I understood the PW process and main parameters of logstash, I was especially interested in thread management, … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1101) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1103)