# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=615 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 616 --- ## [Filter add\_field](https://discuss.elastic.co/t/filter-add-field/164973)
**Author:** [@skylamp](https://discuss.elastic.co/u/skylamp)\ **Replies:** 1\ **Last updated:** [January 21, 2019, 1:06pm UTC](https://discuss.elastic.co/t/filter-add-field/164973 "2019-01-21T13:06:20Z")
Hi I am new to ELK. Need some help with some specific filters. I am fetching some data from a table using a logstash.conf file. I wanted to do this: If status of a user is a or b or c, add a field marking them as X … --- ## [While importing php errors log file in logstash not getting fields seperated from message in elastisearch](https://discuss.elastic.co/t/while-importing-php-errors-log-file-in-logstash-not-getting-fields-seperated-from-message-in-elastisearch/165017)
**Author:** [@Bdr](https://discuss.elastic.co/u/Bdr)\ **Replies:** 1\ **Last updated:** [January 21, 2019, 12:37pm UTC](https://discuss.elastic.co/t/while-importing-php-errors-log-file-in-logstash-not-getting-fields-seperated-from-message-in-elastisearch/165017 "2019-01-21T12:37:20Z")
This is my php error log file \[20-Sep-2018 00:01:00 America/Los\_Angeles\] Uncaught PHP Exception LogicException: "The controller must return a response (null given). Did you forget to add a return statement somewhere in … --- ## [Confused about timezone settings](https://discuss.elastic.co/t/confused-about-timezone-settings/165027)
**Author:** [@fribse](https://discuss.elastic.co/u/fribse)\ **Replies:** 3\ **Last updated:** [January 21, 2019, 11:28am UTC](https://discuss.elastic.co/t/confused-about-timezone-settings/165027 "2019-01-21T11:28:31Z")
Hi Guys I have bunch of servers that has logfiles shown in danish local time, which currently is UTC+1. The filebeat on the server where it harvest the logfile does NOT have a timezone set in the config. In the logsta… --- ## [First attempt into the world of aggregate - string should be a number?](https://discuss.elastic.co/t/first-attempt-into-the-world-of-aggregate-string-should-be-a-number/164312)
**Author:** [@fribse](https://discuss.elastic.co/u/fribse)\ **Replies:** 16\ **Last updated:** [January 21, 2019, 10:07am UTC](https://discuss.elastic.co/t/first-attempt-into-the-world-of-aggregate-string-should-be-a-number/164312 "2019-01-21T10:07:15Z")
So with the help of Badger, I got a working grok filter. Now I need to change it so that it aggregates the lines into one event. One event looks like this 2019-01-12 16:25:01.889 0EF4 \[BERLIN\_PSL -\] Starting to send… --- ## [Logstash csv file to elasticsearch](https://discuss.elastic.co/t/logstash-csv-file-to-elasticsearch/163729)
**Author:** [@ADARSH\_K](https://discuss.elastic.co/u/ADARSH_K)\ **Replies:** 1\ **Last updated:** [January 21, 2019, 10:06am UTC](https://discuss.elastic.co/t/logstash-csv-file-to-elasticsearch/163729 "2019-01-21T10:06:11Z")
I want to insert a .csv file to elasticsearch via logstash.am new in this field and looking in to it more anxiously...pls help --- ## [Logstash + elasticsearch log rotation](https://discuss.elastic.co/t/logstash-elasticsearch-log-rotation/165022)
**Author:** [@HugoSousa](https://discuss.elastic.co/u/HugoSousa)\ **Replies:** 0\ **Last updated:** [January 21, 2019, 9:56am UTC](https://discuss.elastic.co/t/logstash-elasticsearch-log-rotation/165022 "2019-01-21T09:56:33Z")
Hi, Can someone help me with the configuration of logrotation in logstash and in elasticsearch? I am using v5 of elasticsearch. My goal is to have a similar log rotation, like I have configured in filebeat (which is be… --- ## [JDBC password used in pipeline with JDBC input plugin should not be stored as open text in Pipeline](https://discuss.elastic.co/t/jdbc-password-used-in-pipeline-with-jdbc-input-plugin-should-not-be-stored-as-open-text-in-pipeline/163675)
**Author:** [@B123](https://discuss.elastic.co/u/B123)\ **Replies:** 2\ **Last updated:** [January 21, 2019, 8:55am UTC](https://discuss.elastic.co/t/jdbc-password-used-in-pipeline-with-jdbc-input-plugin-should-not-be-stored-as-open-text-in-pipeline/163675 "2019-01-21T08:55:54Z")
HI, Is the keystore feature enabled to store the JDBC password in JDBC input plugin used in logstash pipeline --- ## [Elk architecture](https://discuss.elastic.co/t/elk-architecture/164984)
**Author:** [@sahere37](https://discuss.elastic.co/u/sahere37)\ **Replies:** 2\ **Last updated:** [January 21, 2019, 8:06am UTC](https://discuss.elastic.co/t/elk-architecture/164984 "2019-01-21T08:06:12Z")
hi all, I want to ship different logs located in different machines into a remote machine where one node of elasticsearch installed. for this purpose, i want to install filebeat on each machine where logs are; so accord… --- ## [When importing Drupal watchdog log file in logstash not getting fields separated from message in elastisearch](https://discuss.elastic.co/t/when-importing-drupal-watchdog-log-file-in-logstash-not-getting-fields-separated-from-message-in-elastisearch/163522)
**Author:** [@Bdr](https://discuss.elastic.co/u/Bdr)\ **Replies:** 8\ **Last updated:** [January 21, 2019, 8:00am UTC](https://discuss.elastic.co/t/when-importing-drupal-watchdog-log-file-in-logstash-not-getting-fields-separated-from-message-in-elastisearch/163522 "2019-01-21T08:00:53Z")
This is my watchdog file Jan 3 06:28:34 10.81.157.182 shopingstore: https://secure.shopingstore.com|1546496914|product\_rest|23.194.213.4|https://secure.shopingstore.com/api/product/part?\_format=json||1||Update Part requ… --- ## [How to match multiple field,use multiple field or multiple match or multiple grok](https://discuss.elastic.co/t/how-to-match-multiple-field-use-multiple-field-or-multiple-match-or-multiple-grok/164875)
**Author:** [@hjfeng1988](https://discuss.elastic.co/u/hjfeng1988)\ **Replies:** 2\ **Last updated:** [January 21, 2019, 2:13am UTC](https://discuss.elastic.co/t/how-to-match-multiple-field-use-multiple-field-or-multiple-match-or-multiple-grok/164875 "2019-01-21T02:13:02Z")
Logs file to analyze: GET /login.html?user=hjfeng1988 HTTP/1.1 multiple field filter { grok { match =\> { "message" =\> \["%{WORD:method} %{DATA:request\_uri} HTTP/%{NUMBER:http\_version}"\] "request\_uri… --- ## [Index not created and more it doesn't open and read the log file to ship logs when starting logstash](https://discuss.elastic.co/t/index-not-created-and-more-it-doesnt-open-and-read-the-log-file-to-ship-logs-when-starting-logstash/164598)
**Author:** [@cheriemilk](https://discuss.elastic.co/u/cheriemilk)\ **Replies:** 15\ **Last updated:** [January 20, 2019, 5:31pm UTC](https://discuss.elastic.co/t/index-not-created-and-more-it-doesnt-open-and-read-the-log-file-to-ship-logs-when-starting-logstash/164598 "2019-01-20T17:31:18Z")
I am running to an issue since yesterday that it doesn't open and read the log file when starting logstash. Could you please help take a look what is the problem here? 1. Here is the logstash.conf input { file{… --- ## [Simple logstash.conf not producing index](https://discuss.elastic.co/t/simple-logstash-conf-not-producing-index/164721)
**Author:** [@lynnptilby](https://discuss.elastic.co/u/lynnptilby)\ **Replies:** 4\ **Last updated:** [January 20, 2019, 5:13pm UTC](https://discuss.elastic.co/t/simple-logstash-conf-not-producing-index/164721 "2019-01-20T17:13:01Z")
Here is my logstash.conf: input { file { "path" /home/ubuntu/ELK/2018-12-05-17-33.log } } output { elasticsearch { "hosts" =\> "127.0.0.1" "index" =\> "hwm\_a" } } There are no errors in /var/log/ syslog ./elast… --- ## [What filter in logstash should i use to remove the “\\” from non-Json field](https://discuss.elastic.co/t/what-filter-in-logstash-should-i-use-to-remove-the-from-non-json-field/164645)
**Author:** [@Andrey\_Finkelman](https://discuss.elastic.co/u/Andrey_Finkelman)\ **Replies:** 2\ **Last updated:** [January 20, 2019, 1:34pm UTC](https://discuss.elastic.co/t/what-filter-in-logstash-should-i-use-to-remove-the-from-non-json-field/164645 "2019-01-20T13:34:05Z")
Hi all , i have sending output to Oracle DB , fields containing some XMLs , logstash adding "\\" to all fields as escape character that increasing fields size drasticaly how to remove them from the fields ? Tried : mut… --- ## [Jdbc\_static not looking up](https://discuss.elastic.co/t/jdbc-static-not-looking-up/164553)
**Author:** [@michaeleino](https://discuss.elastic.co/u/michaeleino)\ **Replies:** 2\ **Last updated:** [January 20, 2019, 7:55am UTC](https://discuss.elastic.co/t/jdbc-static-not-looking-up/164553 "2019-01-20T07:55:28Z")
after a long time, getting the plugin starts... it can't resolve the actual values from the derby DB! local\_lookups =\> \[ { id =\> "get-status" query =\> "SELECT name from localstatus WHERE id = :stID" parameters =\> {s… --- ## [Logstash only using 2 core out of 4 core server](https://discuss.elastic.co/t/logstash-only-using-2-core-out-of-4-core-server/164640)
**Author:** [@varun1992](https://discuss.elastic.co/u/varun1992)\ **Replies:** 18\ **Last updated:** [January 19, 2019, 1:36pm UTC](https://discuss.elastic.co/t/logstash-only-using-2-core-out-of-4-core-server/164640 "2019-01-19T13:36:45Z")
Logstash v6.4.3 workers 8 heap size 5gb Logstash only using 2 core out of 4 core system. logstash config file is input { beats { port =\> 9092 } } #---------------------OUTPUT SECTION----------------------… --- ## [Logstash is not running](https://discuss.elastic.co/t/logstash-is-not-running/164878)
**Author:** [@manickalai](https://discuss.elastic.co/u/manickalai)\ **Replies:** 1\ **Last updated:** [January 19, 2019, 12:48pm UTC](https://discuss.elastic.co/t/logstash-is-not-running/164878 "2019-01-19T12:48:43Z")
While Start the logstash getting below error {:timestamp=\>"2019-01-19T11:22:52.600000+0530", :message=\>"Error: Expected one of #, =\> at line 19, column 11 (byte 285) after output {\\n \\telasticsearch {\\n \\t\\thosts ", :l… --- ## [Checking Existence of Sub-Field Revisited](https://discuss.elastic.co/t/checking-existence-of-sub-field-revisited/164862)
**Author:** [@rudyamid](https://discuss.elastic.co/u/rudyamid)\ **Replies:** 4\ **Last updated:** [January 18, 2019, 10:25pm UTC](https://discuss.elastic.co/t/checking-existence-of-sub-field-revisited/164862 "2019-01-18T22:25:01Z")
Looking at previous topics, it seems like the IF-Statement check works, but I haven't been able to prove it in my environment. The logstash conf file looks like this: if !\[system\]\[auth\]\[ssh\]\[ip\] { geoip { sour… --- ## [Input File Path (not working) vs Input Beats (working) all conf setting are the same but Input](https://discuss.elastic.co/t/input-file-path-not-working-vs-input-beats-working-all-conf-setting-are-the-same-but-input/164800)
**Author:** [@Datakids](https://discuss.elastic.co/u/Datakids)\ **Replies:** 5\ **Last updated:** [January 18, 2019, 9:38pm UTC](https://discuss.elastic.co/t/input-file-path-not-working-vs-input-beats-working-all-conf-setting-are-the-same-but-input/164800 "2019-01-18T21:38:45Z")
Hi @ all, I mess arround with follwing and hope you have some Ideas: input { file { path =\> \["C:\\EsKiLo\_6.5.4\\Test.csv"\] start\_position =\> "beginning" } } filter { csv { columns =\>\["Date","Time","… --- ## [Failure\_type\_logging\_whitelist not working for document\_missing\_exception type](https://discuss.elastic.co/t/failure-type-logging-whitelist-not-working-for-document-missing-exception-type/164845)
**Author:** [@jdmcalee](https://discuss.elastic.co/u/jdmcalee)\ **Replies:** 2\ **Last updated:** [January 18, 2019, 7:29pm UTC](https://discuss.elastic.co/t/failure-type-logging-whitelist-not-working-for-document-missing-exception-type/164845 "2019-01-18T19:29:23Z")
I have a pipeline set up to update documents in elasticsearch if they are found to have been removed from a database. Due to implementation details, I've decided it's easier to issue updates that fail if documents don't … --- ## [TCP FIN when using TLS](https://discuss.elastic.co/t/tcp-fin-when-using-tls/164510)
**Author:** [@nich](https://discuss.elastic.co/u/nich)\ **Replies:** 2\ **Last updated:** [January 18, 2019, 5:08pm UTC](https://discuss.elastic.co/t/tcp-fin-when-using-tls/164510 "2019-01-18T17:08:59Z")
We are using Logstash to accept TLS encrypted TCP syslog messages from various devices. Some work fine, but some don't. Looking at a packet capture, we can see that after the TLS handshake, Logstash sends a TCP FIN, just… --- ## [Logstash pipeline error date](https://discuss.elastic.co/t/logstash-pipeline-error-date/164801)
**Author:** [@Jozziej](https://discuss.elastic.co/u/Jozziej)\ **Replies:** 10\ **Last updated:** [January 18, 2019, 4:42pm UTC](https://discuss.elastic.co/t/logstash-pipeline-error-date/164801 "2019-01-18T16:42:24Z")
I'm trying to parse an error log, but my date filter keeps failing example log entry: Wed Nov 28 02:16:24.654 \[MCSS:7497:0x7F17018E7700\]: Warning: invalid cid (-1) in Msg::receive for unknown ip, with msg id (0x2b0300… --- ## [Http\_poller run ruby script and use some vars in the header](https://discuss.elastic.co/t/http-poller-run-ruby-script-and-use-some-vars-in-the-header/164818)
**Author:** [@jehogo](https://discuss.elastic.co/u/jehogo)\ **Replies:** 2\ **Last updated:** [January 18, 2019, 3:55pm UTC](https://discuss.elastic.co/t/http-poller-run-ruby-script-and-use-some-vars-in-the-header/164818 "2019-01-18T15:55:34Z")
I want to retrieve logs from a server using http\_poller plugin but I need to set some vars in the header request. Then, I need to execute the ruby code and use the vars in the input of conf file \*\*api\_key\*\* = '27c59....… --- ## [Failing to import a simple csv](https://discuss.elastic.co/t/failing-to-import-a-simple-csv/164820)
**Author:** [@gbosetti](https://discuss.elastic.co/u/gbosetti)\ **Replies:** 1\ **Last updated:** [January 18, 2019, 3:38pm UTC](https://discuss.elastic.co/t/failing-to-import-a-simple-csv/164820 "2019-01-18T15:38:04Z")
Hello, I'm trying to load some data from a cvs to start learning elastic. But I'm having problems since yesterday and I can't import the data. This is what I did: I downloaded the zipfile and I extracted it. I'm using W… --- ## [Filter - key through values obtained from other keys](https://discuss.elastic.co/t/filter-key-through-values-obtained-from-other-keys/164802)
**Author:** [@Marcos\_OS](https://discuss.elastic.co/u/Marcos_OS)\ **Replies:** 2\ **Last updated:** [January 18, 2019, 3:29pm UTC](https://discuss.elastic.co/t/filter-key-through-values-obtained-from-other-keys/164802 "2019-01-18T15:29:05Z")
Does anyone know if it is possible to mount a key through values obtained from other keys in logstash. My use case: I have data in a relational database in which I am trying to take to elasticsearch through logstash-jd… --- ## [Logstash kafka plugin](https://discuss.elastic.co/t/logstash-kafka-plugin/164814)
**Author:** [@AMITH1284](https://discuss.elastic.co/u/AMITH1284)\ **Replies:** 3\ **Last updated:** [January 18, 2019, 3:21pm UTC](https://discuss.elastic.co/t/logstash-kafka-plugin/164814 "2019-01-18T15:21:53Z")
When sending logs from kafka to ES, if ES is down and logstash fails to send to ES, will it continue to consume messages from kafka or does it stop consuming as it fails to send? Thanks --- ## [Replace timestamp with actual time from the log](https://discuss.elastic.co/t/replace-timestamp-with-actual-time-from-the-log/164813)
**Author:** [@arvind297](https://discuss.elastic.co/u/arvind297)\ **Replies:** 1\ **Last updated:** [January 18, 2019, 3:15pm UTC](https://discuss.elastic.co/t/replace-timestamp-with-actual-time-from-the-log/164813 "2019-01-18T15:15:29Z")
Hi, We are ingesting the log which is in the below format. 2019-01-17T19:01:48.440361Z D \[2027:rrc\_poll\] {t: 3e3dc5ce-0569-11e9-b17d-06a912e3abd4} connection -- update 1 of 1 existing devices, offset 0 2019-01-17T19:… --- ## [How move configuration from logstash 5.5.1 to logstash 6.5 for tomcat logs](https://discuss.elastic.co/t/how-move-configuration-from-logstash-5-5-1-to-logstash-6-5-for-tomcat-logs/164632)
**Author:** [@kvaga](https://discuss.elastic.co/u/kvaga)\ **Replies:** 7\ **Last updated:** [January 18, 2019, 2:17pm UTC](https://discuss.elastic.co/t/how-move-configuration-from-logstash-5-5-1-to-logstash-6-5-for-tomcat-logs/164632 "2019-01-18T14:17:33Z")
Hai, I am a newbie to Elastic Stack. We have a nice working Elastic Stack 5.5 for prod, and I\`m trying to create separate instance on Elastic Stack 6.5.4 for test, and then switch prod to new version No problem with i… --- ## [Filter Array Values](https://discuss.elastic.co/t/filter-array-values/164744)
**Author:** [@ttSS](https://discuss.elastic.co/u/ttSS)\ **Replies:** 7\ **Last updated:** [January 18, 2019, 2:10pm UTC](https://discuss.elastic.co/t/filter-array-values/164744 "2019-01-18T14:10:27Z")
I HAVE AN ARRAY OF MAC ADDRESS THAT I WANT TO FILTER IN LOGSTASH. I want to make it as a new document for each value. Can anybody help me ? {"message":"\["9c:20:7b:25:23:0d","88:d5:0c:cd:e6:09","c8:f2:30:c6:9a:e0","f4:f5… --- ## [Add tag dynamically to create index dynamically](https://discuss.elastic.co/t/add-tag-dynamically-to-create-index-dynamically/164794)
**Author:** [@Roque\_Moyano](https://discuss.elastic.co/u/Roque_Moyano)\ **Replies:** 3\ **Last updated:** [January 18, 2019, 1:40pm UTC](https://discuss.elastic.co/t/add-tag-dynamically-to-create-index-dynamically/164794 "2019-01-18T13:40:43Z")
Hi , I'm trying to create an index for every application that we have in our company , like : index-app1 , index-app2 , we have a lot of team and every team is owner about theirs apps , so I'm trying that approach so be… --- ## [Stalling thread info jdbc](https://discuss.elastic.co/t/stalling-thread-info-jdbc/164624)
**Author:** [@Chivolta](https://discuss.elastic.co/u/Chivolta)\ **Replies:** 3\ **Last updated:** [January 18, 2019, 12:59pm UTC](https://discuss.elastic.co/t/stalling-thread-info-jdbc/164624 "2019-01-18T12:59:41Z")
Hello, Whenever i try to make some changes in my configs, i get: \[2019-01-17T09:04:25,959\]\[INFO \]\[logstash.pipelineaction.reload\] Reloading pipeline {"pipeline.id"=&g\`t;:main} \[2019-01-17T09:04:31,144\]\[WARN \]\[logstas… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=614) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=616)