# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=708 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 709 --- ## [Logstash can't handle quantity of logs?](https://discuss.elastic.co/t/logstash-cant-handle-quantity-of-logs/142925)
**Author:** [@DanV](https://discuss.elastic.co/u/DanV)\ **Replies:** 0\ **Last updated:** [August 3, 2018, 2:25pm UTC](https://discuss.elastic.co/t/logstash-cant-handle-quantity-of-logs/142925 "2018-08-03T14:25:46Z")
Hello all, i'm running logstash 5.6.X in a CentOS 7 virtual machine with 2GB. Logstash is running with a conf.d file that makes it listen to port 5140 (since it can't start a UDP listener on port 514), the logs are bein… --- ## [Getting location out of a json field](https://discuss.elastic.co/t/getting-location-out-of-a-json-field/142893)
**Author:** [@JPelastic](https://discuss.elastic.co/u/JPelastic)\ **Replies:** 1\ **Last updated:** [August 3, 2018, 11:09am UTC](https://discuss.elastic.co/t/getting-location-out-of-a-json-field/142893 "2018-08-03T11:09:24Z")
So i have filters source =\> certificate : which contains a JSON object. Down in this certificate I have a SignatoryLocation. I'm trying to take that SignatoryLocation and add it as a field (so I can use it in a geoip-… --- ## [Is there any work around to get SLK on AIX server](https://discuss.elastic.co/t/is-there-any-work-around-to-get-slk-on-aix-server/142905)
**Author:** [@Ranjith\_sg](https://discuss.elastic.co/u/Ranjith_sg)\ **Replies:** 0\ **Last updated:** [August 3, 2018, 12:13pm UTC](https://discuss.elastic.co/t/is-there-any-work-around-to-get-slk-on-aix-server/142905 "2018-08-03T12:13:06Z")
Hi I'm new to SLK ,could you please let me know is there any workaround to get "SLK" on AIX server .. ? --- ## [Extracting particular folder from the path and adding that to a field](https://discuss.elastic.co/t/extracting-particular-folder-from-the-path-and-adding-that-to-a-field/142277)
**Author:** [@whoiscnu](https://discuss.elastic.co/u/whoiscnu)\ **Replies:** 6\ **Last updated:** [August 3, 2018, 11:49am UTC](https://discuss.elastic.co/t/extracting-particular-folder-from-the-path-and-adding-that-to-a-field/142277 "2018-08-03T11:49:07Z")
Hi Guys, I need help to fetch particular folder from the path and assign the same to a different field. path ==\> /var/mqm/qmgrs/FOLDER\_NEEDED/errors/\*.log. I need to extract FOLDER\_NEEDED and assign it to a field for … --- ## [Multiline: flush downstream error. NullPointerException](https://discuss.elastic.co/t/multiline-flush-downstream-error-nullpointerexception/140920)
**Author:** [@hurg](https://discuss.elastic.co/u/hurg)\ **Replies:** 1\ **Last updated:** [August 3, 2018, 11:32am UTC](https://discuss.elastic.co/t/multiline-flush-downstream-error-nullpointerexception/140920 "2018-08-03T11:32:07Z")
Hey guys, did someone get the similar errors? I found a few topics here but they are not helpful. Logstash version is 5.2.2 14:45:11.380 \[nioEventLoopGroup-4-1\] ERROR logstash.codecs.multiline - Multiline: flush downst… --- ## [Logstash aborting due to Pipeline Configuration Error](https://discuss.elastic.co/t/logstash-aborting-due-to-pipeline-configuration-error/142347)
**Author:** [@rshetty](https://discuss.elastic.co/u/rshetty)\ **Replies:** 6\ **Last updated:** [August 3, 2018, 11:06am UTC](https://discuss.elastic.co/t/logstash-aborting-due-to-pipeline-configuration-error/142347 "2018-08-03T11:06:37Z")
I have multiple pipelines running in my logstash server that have been configured via kibana. However, I'm facing an issue where if one of the pipeline configurations has some error than the entire logstash instance abor… --- ## [Multiple pipelines + docker](https://discuss.elastic.co/t/multiple-pipelines-docker/141728)
**Author:** [@hilt86](https://discuss.elastic.co/u/hilt86)\ **Replies:** 2\ **Last updated:** [August 3, 2018, 10:31am UTC](https://discuss.elastic.co/t/multiple-pipelines-docker/141728 "2018-08-03T10:31:59Z")
After reading the docs on running logstash on docker it isn't clear how we should treat the pipelines.yml file when using docker. I have tried bind mounting like so :slight\_smile: volumes: - /root/pipeline2/:/usr/… --- ## [Importing shinken logs into Elasticsearch with logstash](https://discuss.elastic.co/t/importing-shinken-logs-into-elasticsearch-with-logstash/139758)
**Author:** [@Goz](https://discuss.elastic.co/u/Goz)\ **Replies:** 4\ **Last updated:** [August 3, 2018, 9:43am UTC](https://discuss.elastic.co/t/importing-shinken-logs-into-elasticsearch-with-logstash/139758 "2018-08-03T09:43:12Z")
Hi I'm installing shinken to make the monitoring and I wish to import the shinken logs into elasticsearch with logstash. someone can help me. Thank you for your help. --- ## [Logstash error while using email plugin](https://discuss.elastic.co/t/logstash-error-while-using-email-plugin/142840)
**Author:** [@APJ](https://discuss.elastic.co/u/APJ)\ **Replies:** 16\ **Last updated:** [August 3, 2018, 8:09am UTC](https://discuss.elastic.co/t/logstash-error-while-using-email-plugin/142840 "2018-08-03T08:09:02Z")
I am trying to simulate an environment wherein if the temperature exceeds 40℉, an email is sent to the user. I am running logstash on my local. Here is the config file. weather\_data.conf input { http\_poller { ur… --- ## [SQL query from Logstash causes an error PGobject](https://discuss.elastic.co/t/sql-query-from-logstash-causes-an-error-pgobject/138648)
**Author:** [@Igorr](https://discuss.elastic.co/u/Igorr)\ **Replies:** 1\ **Last updated:** [July 9, 2018, 11:36am UTC](https://discuss.elastic.co/t/sql-query-from-logstash-causes-an-error-pgobject/138648 "2018-07-09T11:36:02Z")
Logstash passes a request type: SELECT \* from dbmon\_stat\_activity() I get an error in the log: \_\[2018-07-02T14:00:00,384\]\[WARN \]\[logstash.inputs.jdbc \] Exception when executing JDBC query {:exception=\>#\ **Author:** [@prashansa1](https://discuss.elastic.co/u/prashansa1)\ **Replies:** 3\ **Last updated:** [August 3, 2018, 6:24am UTC](https://discuss.elastic.co/t/how-to-modify-the-value-of-exported-variable-in-logtash-file/142841 "2018-08-03T06:24:17Z") I have one variable in shell script and exported it and I want to increase( by 1) the value of this variable in filter block of logstash file . I have one variable in shell script (ksh file) export f=0 logstash file … --- ## [Logstash \_dateparsefailure](https://discuss.elastic.co/t/logstash-dateparsefailure/142795)
**Author:** [@Sebastian\_Herrera](https://discuss.elastic.co/u/Sebastian_Herrera)\ **Replies:** 3\ **Last updated:** [August 3, 2018, 4:57am UTC](https://discuss.elastic.co/t/logstash-dateparsefailure/142795 "2018-08-03T04:57:01Z")
Hello, i am trying to use date filter but i am getting \_dateparsefailure can anyone help me? Log: { "timestamp": "Aug 2 10:47:59", "@timestamp": "2018-08-02T17:47:59.000Z", "timezone": "US/Pacific", "tags": \[ "sy… --- ## [Logstash creates a new index every 24 hours](https://discuss.elastic.co/t/logstash-creates-a-new-index-every-24-hours/140793)
**Author:** [@APJ](https://discuss.elastic.co/u/APJ)\ **Replies:** 15\ **Last updated:** [August 3, 2018, 4:04am UTC](https://discuss.elastic.co/t/logstash-creates-a-new-index-every-24-hours/140793 "2018-08-03T04:04:17Z")
Hi all, I am using https://openweathermap.org/ current weather API to display current weather data. I was able successfully visualize the data on Kibana after it created a new index weather-2018.07.18. Here is my logs… --- ## [How to use grok to filter some fields in message in logstash, skip the unnecessary, only filter the specified](https://discuss.elastic.co/t/how-to-use-grok-to-filter-some-fields-in-message-in-logstash-skip-the-unnecessary-only-filter-the-specified/142754)
**Author:** [@sun\_changlong](https://discuss.elastic.co/u/sun_changlong)\ **Replies:** 5\ **Last updated:** [August 3, 2018, 3:54am UTC](https://discuss.elastic.co/t/how-to-use-grok-to-filter-some-fields-in-message-in-logstash-skip-the-unnecessary-only-filter-the-specified/142754 "2018-08-03T03:54:37Z")
For example, the data format is as follows: dev="1235667" pri="6" dev\_type="ddos\_clean" recorder="ads" vsid="0" sub\_type=attacklog dst\_addr=1.1.1.1 zonename=1504152157558 grpname=test Just get the contents of the sub… --- ## [Heavy Traffic on Logstash causing data inconsistencies](https://discuss.elastic.co/t/heavy-traffic-on-logstash-causing-data-inconsistencies/142839)
**Author:** [@keeshqs](https://discuss.elastic.co/u/keeshqs)\ **Replies:** 0\ **Last updated:** [August 3, 2018, 3:34am UTC](https://discuss.elastic.co/t/heavy-traffic-on-logstash-causing-data-inconsistencies/142839 "2018-08-03T03:34:59Z")
Hi all, We would like to seek for your advice regarding an issue we have on Logstash. Currently we are receiving data from 18 servers/clients (each with 9 check metrics that creates 9 indices in ES per day). The event d… --- ## [Grok Printer Log issue](https://discuss.elastic.co/t/grok-printer-log-issue/142697)
**Author:** [@Pinno\_Lin](https://discuss.elastic.co/u/Pinno_Lin)\ **Replies:** 5\ **Last updated:** [August 3, 2018, 3:11am UTC](https://discuss.elastic.co/t/grok-printer-log-issue/142697 "2018-08-03T03:11:45Z")
Hi, I want to grok the Printer Log and provide 10 records sample are below. ABC00720,ABC00720\_NB,TCPRINT501,5,1491672,A4,Microsoft Outlook - 備忘錄樣式,2018/08/01,22:42:42 ABC03951,ABC03951\_NB,RDPRINT501,6,545477,Letter (8… --- ## [Logstash windows version 6.3.2 wont start](https://discuss.elastic.co/t/logstash-windows-version-6-3-2-wont-start/141664)
**Author:** [@jiawang](https://discuss.elastic.co/u/jiawang)\ **Replies:** 5\ **Last updated:** [August 2, 2018, 10:24pm UTC](https://discuss.elastic.co/t/logstash-windows-version-6-3-2-wont-start/141664 "2018-08-02T22:24:36Z")
I am new to elastic. Just downloaded the version of logstash 6.3.2, and want to try it with other components (beat, elasticsearch, kibana). So far, beat, elasticsearch and kibanna are working. However, when i followed th… --- ## [Storing username and password somewhere and somehow](https://discuss.elastic.co/t/storing-username-and-password-somewhere-and-somehow/142220)
**Author:** [@lucas.alvarez](https://discuss.elastic.co/u/lucas.alvarez)\ **Replies:** 5\ **Last updated:** [August 2, 2018, 6:35pm UTC](https://discuss.elastic.co/t/storing-username-and-password-somewhere-and-somehow/142220 "2018-08-02T18:35:53Z")
Hello everybody!. I would like to know how/where I can store credentials like username and password to be consumed from the output.conf file. Any ideas? --- ## [Is there a way to alter the value of live API data polled using http\_poller input plugin?](https://discuss.elastic.co/t/is-there-a-way-to-alter-the-value-of-live-api-data-polled-using-http-poller-input-plugin/142671)
**Author:** [@APJ](https://discuss.elastic.co/u/APJ)\ **Replies:** 5\ **Last updated:** [August 2, 2018, 5:57pm UTC](https://discuss.elastic.co/t/is-there-a-way-to-alter-the-value-of-live-api-data-polled-using-http-poller-input-plugin/142671 "2018-08-02T17:57:19Z")
Hi all, I have a logstash config file that pulls in live data using the http\_poller plugin. Here is my logstash.conf input { http\_poller { urls =\> { weather =\> { url =\> "http://api.openweather… --- ## [How to set up input for https curl and header key?](https://discuss.elastic.co/t/how-to-set-up-input-for-https-curl-and-header-key/142231)
**Author:** [@emanresu](https://discuss.elastic.co/u/emanresu)\ **Replies:** 15\ **Last updated:** [August 2, 2018, 4:34pm UTC](https://discuss.elastic.co/t/how-to-set-up-input-for-https-curl-and-header-key/142231 "2018-08-02T16:34:24Z")
I need to invoke a streaming endpoint by executing the following: curl -X GET "https://feed.source.com/1.0/json/A9575EEEE9F53398FD237049" -H "api\_key:zBA7goakSWiE1A7aHA" I tried this input { http { host =\> "https… --- ## [Can't create a field with a variable from a grok match regex](https://discuss.elastic.co/t/cant-create-a-field-with-a-variable-from-a-grok-match-regex/142613)
**Author:** [@TheSmartMonkey](https://discuss.elastic.co/u/TheSmartMonkey)\ **Replies:** 6\ **Last updated:** [August 2, 2018, 3:18pm UTC](https://discuss.elastic.co/t/cant-create-a-field-with-a-variable-from-a-grok-match-regex/142613 "2018-08-02T15:18:00Z")
Hi all, I am currently using logstash, elasticsearch and kibana 6.3.0 My log are generated at a unique id path : /tmp/USER\_DATA/FactoryContainer/images/(my unique id)/oar/oar\_image\_job(my unique id).stdout What i want… --- ## [Repetitive JDBC input runs](https://discuss.elastic.co/t/repetitive-jdbc-input-runs/142257)
**Author:** [@daq](https://discuss.elastic.co/u/daq)\ **Replies:** 7\ **Last updated:** [August 2, 2018, 3:01pm UTC](https://discuss.elastic.co/t/repetitive-jdbc-input-runs/142257 "2018-08-02T15:01:23Z")
We have a stored procedure which returns data for the last hour when called without any parameters. We scheduled the JDBC input plugin to run every hour and it works well. Questions is, if I want to ingest older data I c… --- ## [Logstash-output-syslog full json in message?](https://discuss.elastic.co/t/logstash-output-syslog-full-json-in-message/142642)
**Author:** [@PandKing](https://discuss.elastic.co/u/PandKing)\ **Replies:** 10\ **Last updated:** [August 2, 2018, 2:58pm UTC](https://discuss.elastic.co/t/logstash-output-syslog-full-json-in-message/142642 "2018-08-02T14:58:30Z")
Hi, I'm sending filebeat data to logstash and from there I send a copy to elastic and another to a 3rd party SIM which only supports syslog. Is there a way to send the full json line in the message without adding a bun… --- ## [Logging options in logstash.yml are not working](https://discuss.elastic.co/t/logging-options-in-logstash-yml-are-not-working/142773)
**Author:** [@twan](https://discuss.elastic.co/u/twan)\ **Replies:** 4\ **Last updated:** [August 2, 2018, 2:57pm UTC](https://discuss.elastic.co/t/logging-options-in-logstash-yml-are-not-working/142773 "2018-08-02T14:57:24Z")
Hi there! We want Logstash to send its logging to a specified location. If we set (the data folder is mounted external from the docker): log.level: warn path.logs: /usr/share/logstash/data/log It does not create a f… --- ## [Special occurences](https://discuss.elastic.co/t/special-occurences/142652)
**Author:** [@dersoi](https://discuss.elastic.co/u/dersoi)\ **Replies:** 6\ **Last updated:** [August 2, 2018, 2:43pm UTC](https://discuss.elastic.co/t/special-occurences/142652 "2018-08-02T14:43:37Z")
Hi, I use my ELK stack stock logs, and sometimes, logs will be in this format field1, field2, field3, and so on, so I basically CSV to separate the fields. A little problem rose... Sometimes, the field will actually b… --- ## [Logstash file-input problem](https://discuss.elastic.co/t/logstash-file-input-problem/142395)
**Author:** [@sivap](https://discuss.elastic.co/u/sivap)\ **Replies:** 5\ **Last updated:** [August 2, 2018, 2:09pm UTC](https://discuss.elastic.co/t/logstash-file-input-problem/142395 "2018-08-02T14:09:58Z")
Hi Team, Every 5 minutes once my cron job will create one new text file ( filename with time - jobs\_00:00.txt, jobs\_00:05.txt, jobs\_00:10.txt, jobs\_00:15.txt, and so on..). So I have created logstash config using file-… --- ## [Grok parsing syslog error](https://discuss.elastic.co/t/grok-parsing-syslog-error/142748)
**Author:** [@DanielE](https://discuss.elastic.co/u/DanielE)\ **Replies:** 3\ **Last updated:** [August 2, 2018, 1:48pm UTC](https://discuss.elastic.co/t/grok-parsing-syslog-error/142748 "2018-08-02T13:48:26Z")
Hey there, i am trying to gather my syslog infos parse them trough a grok filter. But i am getting \_grokparsefailure\_sysloginput and \_grokparsefailure. I am using the following configs: input input { syslog { … --- ## [How to Parse logfile with columns informations](https://discuss.elastic.co/t/how-to-parse-logfile-with-columns-informations/142699)
**Author:** [@Loudone](https://discuss.elastic.co/u/Loudone)\ **Replies:** 7\ **Last updated:** [August 2, 2018, 10:54am UTC](https://discuss.elastic.co/t/how-to-parse-logfile-with-columns-informations/142699 "2018-08-02T10:54:50Z")
Dear All, I'm trying to parse a very specific Log, this log look like this: pid user\_name user\_authentication client\_host client\_lib\_ver === ========= =================== =========== ============== 510 dmadmi… --- ## [Assigning the date in a CSV column to the timestamp](https://discuss.elastic.co/t/assigning-the-date-in-a-csv-column-to-the-timestamp/142727)
**Author:** [@kolli\_dilip](https://discuss.elastic.co/u/kolli_dilip)\ **Replies:** 3\ **Last updated:** [August 2, 2018, 9:55am UTC](https://discuss.elastic.co/t/assigning-the-date-in-a-csv-column-to-the-timestamp/142727 "2018-08-02T09:55:20Z")
Hi All, Need assistance in assigning the date in a CSV column to the timestamp The sample set of the field is as below The config used is as below and the output is as below in stdout console, where the data is n… --- ## [Logstash JDBC input: Encrypted username and password](https://discuss.elastic.co/t/logstash-jdbc-input-encrypted-username-and-password/142538)
**Author:** [@Goyal.vikas87](https://discuss.elastic.co/u/Goyal.vikas87)\ **Replies:** 2\ **Last updated:** [August 2, 2018, 9:43am UTC](https://discuss.elastic.co/t/logstash-jdbc-input-encrypted-username-and-password/142538 "2018-08-02T09:43:56Z")
Please help how I can use encrypted username and password in logstash config files rather than hard coding in input section. Is there any way that I can read the parameters from an external file. [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=707) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=709)