# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=568 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 569 --- ## [Logstash adds timestamp value to the output logs automatically](https://discuss.elastic.co/t/logstash-adds-timestamp-value-to-the-output-logs-automatically/176251)
**Author:** [@naveenrt23](https://discuss.elastic.co/u/naveenrt23)\ **Replies:** 0\ **Last updated:** [April 10, 2019, 3:10pm UTC](https://discuss.elastic.co/t/logstash-adds-timestamp-value-to-the-output-logs-automatically/176251 "2019-04-10T15:10:17Z")
Hello, I am trying to use influxdb output plugin v-5.0.5 with logstash v-6.2.3. I am getting parsing exceptions but on further looking,i see "logstash and a timestamp value" is getting added to all the output logs. Whe… --- ## [Update ElasticSearch existing documents with new fields](https://discuss.elastic.co/t/update-elasticsearch-existing-documents-with-new-fields/176051)
**Author:** [@jcaballero](https://discuss.elastic.co/u/jcaballero)\ **Replies:** 1\ **Last updated:** [April 10, 2019, 1:45pm UTC](https://discuss.elastic.co/t/update-elasticsearch-existing-documents-with-new-fields/176051 "2019-04-10T13:45:31Z")
Hello, FileBeat 6.6.2 LogStash 6.6.2 ElasticSearch 5.6.10 I am trying to combine data from 2 different log files into the same document in ElasticSearch. I read each log file with a separate instance of FileBeat, bo… --- ## [.NET-specific: PUT requests to LogStash fail when sent using HttpClient, succeed when sent using cURL](https://discuss.elastic.co/t/net-specific-put-requests-to-logstash-fail-when-sent-using-httpclient-succeed-when-sent-using-curl/176209)
**Author:** [@Miao](https://discuss.elastic.co/u/Miao)\ **Replies:** 1\ **Last updated:** [April 10, 2019, 1:31pm UTC](https://discuss.elastic.co/t/net-specific-put-requests-to-logstash-fail-when-sent-using-httpclient-succeed-when-sent-using-curl/176209 "2019-04-10T13:31:15Z")
Here is my logstash.conf file: input { http { host =\> "127.0.0.1" port =\> 31311 ssl =\> false } } filter { mutate { split =\> { "\[headers\]\[request\_path\]" =\> "/"} add\_field =\> { "index\_id" =\> "%{\[headers\]\[re… --- ## [JSON parse error on valid JSON](https://discuss.elastic.co/t/json-parse-error-on-valid-json/176221)
**Author:** [@Miao](https://discuss.elastic.co/u/Miao)\ **Replies:** 1\ **Last updated:** [April 10, 2019, 1:25pm UTC](https://discuss.elastic.co/t/json-parse-error-on-valid-json/176221 "2019-04-10T13:25:42Z")
Here is my logstash.conf file: input { http { host =\> "127.0.0.1" port =\> 31311 ssl =\> false } } filter { mutate { split =\> { "\[headers\]\[request\_path\]" =\> "/"} add\_field =\> { "index\_id" =\> "%{\[headers\]\[re… --- ## [Logstash Grok filter query in Logstash.conf](https://discuss.elastic.co/t/logstash-grok-filter-query-in-logstash-conf/176195)
**Author:** [@Raju\_Bhagat](https://discuss.elastic.co/u/Raju_Bhagat)\ **Replies:** 1\ **Last updated:** [April 10, 2019, 12:22pm UTC](https://discuss.elastic.co/t/logstash-grok-filter-query-in-logstash-conf/176195 "2019-04-10T12:22:47Z")
Dear Community Members, I am new to grok. Following are the log message to which I want to extract fields values like : Method : createCabinet Argument : {"comments":....} Result : \<201 Created,Folder .... Time : … --- ## [SSL error received even though "ssl" is set to "false"](https://discuss.elastic.co/t/ssl-error-received-even-though-ssl-is-set-to-false/176163)
**Author:** [@Miao](https://discuss.elastic.co/u/Miao)\ **Replies:** 3\ **Last updated:** [April 10, 2019, 11:51am UTC](https://discuss.elastic.co/t/ssl-error-received-even-though-ssl-is-set-to-false/176163 "2019-04-10T11:51:04Z")
Here is the input segment of my logstash.conf: input { http { host =\> "127.0.0.1" port =\> 31311 ssl =\> false } } I have an index named twitter\_new. I want to send a DELETE request via cURL to remove the entire… --- ## [How can use property ca\_file in Elasticsearch input plugin?](https://discuss.elastic.co/t/how-can-use-property-ca-file-in-elasticsearch-input-plugin/176205)
**Author:** [@zwk](https://discuss.elastic.co/u/zwk)\ **Replies:** 0\ **Last updated:** [April 10, 2019, 11:46am UTC](https://discuss.elastic.co/t/how-can-use-property-ca-file-in-elasticsearch-input-plugin/176205 "2019-04-10T11:46:18Z")
When I use ca\_file in Elasticsearch input plugin, A error apper. The config is : input { elasticsearch { hosts =\> "elasticsearch:9200" index =\> "index-2019-04-07" size =\> 1000 scroll =\> "5m" docinfo =\> true ca\_f… --- ## [How to use the input kafka topic in the output elasticsearch with logstash?](https://discuss.elastic.co/t/how-to-use-the-input-kafka-topic-in-the-output-elasticsearch-with-logstash/176186)
**Author:** [@evahsiao](https://discuss.elastic.co/u/evahsiao)\ **Replies:** 2\ **Last updated:** [April 10, 2019, 11:17am UTC](https://discuss.elastic.co/t/how-to-use-the-input-kafka-topic-in-the-output-elasticsearch-with-logstash/176186 "2019-04-10T11:17:25Z")
Here is my use case: I use kafka messages is the input data and then output to the elasticsearch database. I consume multi topics as below setting: input { kafka { bootstrap\_servers =\> "127.0.0.1:9092" g… --- ## [Issue with custom parsing in grok filter](https://discuss.elastic.co/t/issue-with-custom-parsing-in-grok-filter/172351)
**Author:** [@jprakash402](https://discuss.elastic.co/u/jprakash402)\ **Replies:** 2\ **Last updated:** [April 10, 2019, 11:08am UTC](https://discuss.elastic.co/t/issue-with-custom-parsing-in-grok-filter/172351 "2019-04-10T11:08:20Z")
Hi All, We have a custom log file, which we need to parse to extract some required fields out of it. We have setup Logstash to parse the data and push it to Elasticsearch. Sample log file. {"level":"info","message":"AP… --- ## [HowTo: Integer comparison in logstash filter](https://discuss.elastic.co/t/howto-integer-comparison-in-logstash-filter/175675)
**Author:** [@arberg](https://discuss.elastic.co/u/arberg)\ **Replies:** 2\ **Last updated:** [April 10, 2019, 10:51am UTC](https://discuss.elastic.co/t/howto-integer-comparison-in-logstash-filter/175675 "2019-04-10T10:51:14Z")
I cannot get integer comparisons to work in logstash filter, using logstash 6.4.2. input { beats { port =\> 5044 } } filter { mutate { add\_field =\> { "\[@metadata\]\[myLogLevel\]" =\> 1 } add\_fiel… --- ## [JSON Array by index](https://discuss.elastic.co/t/json-array-by-index/176057)
**Author:** [@Jacob-N](https://discuss.elastic.co/u/Jacob-N)\ **Replies:** 2\ **Last updated:** [April 10, 2019, 10:25am UTC](https://discuss.elastic.co/t/json-array-by-index/176057 "2019-04-10T10:25:57Z")
I am parsing a JSON from http\_poller. The JSON contains an array of coordinates with 3 numbers in the array. Only the first 2 are the lat and longitude and I would like to discard the 3rd number for the time being. Is it… --- ## [How to remove fields in logstash output](https://discuss.elastic.co/t/how-to-remove-fields-in-logstash-output/173469)
**Author:** [@evahsiao](https://discuss.elastic.co/u/evahsiao)\ **Replies:** 7\ **Last updated:** [April 10, 2019, 9:27am UTC](https://discuss.elastic.co/t/how-to-remove-fields-in-logstash-output/173469 "2019-04-10T09:27:03Z")
I want to pass my json data from kafka to elasticseaarch. Judged by the operation param to different action of elasticsearch. my json data is like this: { "operation": "create", "fields": { "sign": "im\_messages", "… --- ## [Unable to consume a message from ActiveMQ topic](https://discuss.elastic.co/t/unable-to-consume-a-message-from-activemq-topic/175763)
**Author:** [@rajeshk150](https://discuss.elastic.co/u/rajeshk150)\ **Replies:** 2\ **Last updated:** [April 10, 2019, 7:14am UTC](https://discuss.elastic.co/t/unable-to-consume-a-message-from-activemq-topic/175763 "2019-04-10T07:14:17Z")
Hi, I am unable to consume a message from activemq topic. The topic with the given name will be created in activemq but when I send a message for that topic, I am unable to consume the same message in logstash. The acti… --- ## [Grok filter TIMESTAMP\_ISO8601 Logstash not working](https://discuss.elastic.co/t/grok-filter-timestamp-iso8601-logstash-not-working/175812)
**Author:** [@hatienkma](https://discuss.elastic.co/u/hatienkma)\ **Replies:** 2\ **Last updated:** [April 10, 2019, 7:12am UTC](https://discuss.elastic.co/t/grok-filter-timestamp-iso8601-logstash-not-working/175812 "2019-04-10T07:12:39Z")
Hello, I have problem and need help. This is my logfile type: 2019-01-28 07:37:25 52.62.158.16 - HTTP 192.168.25.80 443 GET /cpadmin/jquery-ui.min.js - 304 573 197 0 HTTP/1.1 Mozilla/5.0+(Windows+NT+10.0;+WOW64)+Apple… --- ## [Regarding creating index pattern for .txt log files](https://discuss.elastic.co/t/regarding-creating-index-pattern-for-txt-log-files/175979)
**Author:** [@Kalyan\_MB](https://discuss.elastic.co/u/Kalyan_MB)\ **Replies:** 11\ **Last updated:** [April 10, 2019, 7:00am UTC](https://discuss.elastic.co/t/regarding-creating-index-pattern-for-txt-log-files/175979 "2019-04-10T07:00:58Z")
Hello, i have below requirement to achieve can anyone help me with this? 1.Need to scan performance logs from a directory(client), and parse them to ELK stack(server) create index patterns using lostash.conf file fo… --- ## [Index is not got created into elastic search](https://discuss.elastic.co/t/index-is-not-got-created-into-elastic-search/176053)
**Author:** [@amit\_anand](https://discuss.elastic.co/u/amit_anand)\ **Replies:** 3\ **Last updated:** [April 10, 2019, 5:48am UTC](https://discuss.elastic.co/t/index-is-not-got-created-into-elastic-search/176053 "2019-04-10T05:48:01Z")
This is the config file i m using to run logstatsh and version is 6.7.1 but when i try to search into kabana its not showing any indices with "mchoutput" name. Please help me ..Thanks in advance.. input { file { pat… --- ## [Multiple file setting in input in logstash conf](https://discuss.elastic.co/t/multiple-file-setting-in-input-in-logstash-conf/176084)
**Author:** [@inandi](https://discuss.elastic.co/u/inandi)\ **Replies:** 2\ **Last updated:** [April 10, 2019, 1:17am UTC](https://discuss.elastic.co/t/multiple-file-setting-in-input-in-logstash-conf/176084 "2019-04-10T01:17:02Z")
my ELK version 6.7. i want to add multiple files in logstash conf file and use it conditinally. i have this conf file but it is not working. input { file { type =\> "back" path =\> "E:/xampp/htdocs/project/hq/l… --- ## [Logstash cant connect to Elasticsearch](https://discuss.elastic.co/t/logstash-cant-connect-to-elasticsearch/176107)
**Author:** [@Savagepony](https://discuss.elastic.co/u/Savagepony)\ **Replies:** 1\ **Last updated:** [April 9, 2019, 9:32pm UTC](https://discuss.elastic.co/t/logstash-cant-connect-to-elasticsearch/176107 "2019-04-09T21:32:38Z")
I am getting the below error when I am starting the logstash \[2019-04-09T17:19:18,094\]\[WARN \]\[logstash.outputs.elasticsearch\] Attempted to resurrect connection to dead ES instance, but got an error. {:url=\>"https://loca… --- ## [Error while sending data to amazon\_es](https://discuss.elastic.co/t/error-while-sending-data-to-amazon-es/176074)
**Author:** [@Shubham\_Yadav1](https://discuss.elastic.co/u/Shubham_Yadav1)\ **Replies:** 1\ **Last updated:** [April 9, 2019, 7:08pm UTC](https://discuss.elastic.co/t/error-while-sending-data-to-amazon-es/176074 "2019-04-09T19:08:13Z")
i have a config file which is working fine on my localhost but while sending same data to amazon\_es i encounter this error Failed to install template: \[400\] {"error":{"root\_cause":\[{"type":"mapper\_parsing\_exception","re… --- ## [Convert a field value from Decimal to Hexa](https://discuss.elastic.co/t/convert-a-field-value-from-decimal-to-hexa/176054)
**Author:** [@WarriorHarb](https://discuss.elastic.co/u/WarriorHarb)\ **Replies:** 6\ **Last updated:** [April 9, 2019, 4:51pm UTC](https://discuss.elastic.co/t/convert-a-field-value-from-decimal-to-hexa/176054 "2019-04-09T16:51:32Z")
Hello, So after applying the GROK filter ,i have a field that it's value is in decimal. I need to convert it to Hexa. i tried this : ruby { code =\> "event.set('%{AgentID}', event.get('%{AgentID}').to\_s.hex)" } t… --- ## [Logstash: Could not index event to Elasticsearch](https://discuss.elastic.co/t/logstash-could-not-index-event-to-elasticsearch/176066)
**Author:** [@ep4sh](https://discuss.elastic.co/u/ep4sh)\ **Replies:** 0\ **Last updated:** [April 9, 2019, 3:47pm UTC](https://discuss.elastic.co/t/logstash-could-not-index-event-to-elasticsearch/176066 "2019-04-09T15:47:42Z")
I have many many errors in Logstash log: \[2019-04-09T11:24:48,037\]\[WARN \]\[logstash.outputs.elasticsearch\] Could not index event to Elasticsearch. {:status=\>400, :action=\>\["index", {:\_id=\>nil, :\_index=\>"applicationlog-n… --- ## [Logstash-input-mongodb (questions)](https://discuss.elastic.co/t/logstash-input-mongodb-questions/173363)
**Author:** [@pmarques](https://discuss.elastic.co/u/pmarques)\ **Replies:** 5\ **Last updated:** [April 9, 2019, 3:09pm UTC](https://discuss.elastic.co/t/logstash-input-mongodb-questions/173363 "2019-04-09T15:09:58Z")
Hello All, I have read some old discussions about using MongoDB as a data source. I mean, using Logstash to bring data from MongoDB and send it to Elasticsearch. Basically, this (https://discuss.elastic.co/t/mongodb-log… --- ## [Parsing result logs in double](https://discuss.elastic.co/t/parsing-result-logs-in-double/176038)
**Author:** [@hzarrabi](https://discuss.elastic.co/u/hzarrabi)\ **Replies:** 0\ **Last updated:** [April 9, 2019, 1:57pm UTC](https://discuss.elastic.co/t/parsing-result-logs-in-double/176038 "2019-04-09T13:57:11Z")
Hi, Does anyone encounters double logs parsed by Logstash. Indeed, it happens sometimes and not always that the entire log file is parsed two times. (Each log appears two times in Kibana.) Sincedb is healthy log file… --- ## [Pull values from array of json](https://discuss.elastic.co/t/pull-values-from-array-of-json/175900)
**Author:** [@Michael\_MacPherson](https://discuss.elastic.co/u/Michael_MacPherson)\ **Replies:** 7\ **Last updated:** [April 9, 2019, 1:35pm UTC](https://discuss.elastic.co/t/pull-values-from-array-of-json/175900 "2019-04-09T13:35:23Z")
Hi, I have the following array as a field in my event and I am looking to pull out the value from name as a list. For example: {colour=#f26211, exportable=true, numerical\_value=null, hide\_tag=false, user\_id=0, name=ta… --- ## [Pipelines.yml and best practices using this approach (+auto config reload)](https://discuss.elastic.co/t/pipelines-yml-and-best-practices-using-this-approach-auto-config-reload/175824)
**Author:** [@saif3r](https://discuss.elastic.co/u/saif3r)\ **Replies:** 4\ **Last updated:** [April 9, 2019, 1:32pm UTC](https://discuss.elastic.co/t/pipelines-yml-and-best-practices-using-this-approach-auto-config-reload/175824 "2019-04-09T13:32:46Z")
Hello Guys, I just created several different pipelines and I'm running them using pipelines.yml and systemctl command to control logstash service. I have a few questions regarding this approach, I hope you can answer th… --- ## [Logstash parser to read multiple files through input and process](https://discuss.elastic.co/t/logstash-parser-to-read-multiple-files-through-input-and-process/175943)
**Author:** [@tkb](https://discuss.elastic.co/u/tkb)\ **Replies:** 2\ **Last updated:** [April 9, 2019, 12:46pm UTC](https://discuss.elastic.co/t/logstash-parser-to-read-multiple-files-through-input-and-process/175943 "2019-04-09T12:46:26Z")
Hi - I've testng xml file contains testcaseName, status(pass/fail), error Desc, etc And have another file jenkins.txt contains only one value jenkinBuildNumber=35 Now my requirement is to read these 2 files and add… --- ## [Document\_id not set in ElasticSearch, even though it is configured inside logstash.conf](https://discuss.elastic.co/t/document-id-not-set-in-elasticsearch-even-though-it-is-configured-inside-logstash-conf/176016)
**Author:** [@Miao](https://discuss.elastic.co/u/Miao)\ **Replies:** 4\ **Last updated:** [April 9, 2019, 12:37pm UTC](https://discuss.elastic.co/t/document-id-not-set-in-elasticsearch-even-though-it-is-configured-inside-logstash-conf/176016 "2019-04-09T12:37:17Z")
Here are the contents inside my logstash.conf: input { http { host =\> "127.0.0.1" port =\> 31311 } } filter { mutate { split =\> { "\[headers\]\[request\_path\]" =\> "/"} add\_field =\> { "index\_id" =\> "%{\[headers\]\[r… --- ## [Gap in basic understanding of message processing of Logstash](https://discuss.elastic.co/t/gap-in-basic-understanding-of-message-processing-of-logstash/175980)
**Author:** [@saif3r](https://discuss.elastic.co/u/saif3r)\ **Replies:** 9\ **Last updated:** [April 9, 2019, 12:36pm UTC](https://discuss.elastic.co/t/gap-in-basic-understanding-of-message-processing-of-logstash/175980 "2019-04-09T12:36:35Z")
Hi Guys, I did a quick training about Logstash on Udemy and I think i forgot about one important thing and didn't took a note. So I could use a brief explanation on what do I do wrong. Based on the message content, I w… --- ## [Logstash output not the same as grok debugger](https://discuss.elastic.co/t/logstash-output-not-the-same-as-grok-debugger/175258)
**Author:** [@markov](https://discuss.elastic.co/u/markov)\ **Replies:** 12\ **Last updated:** [April 9, 2019, 12:24pm UTC](https://discuss.elastic.co/t/logstash-output-not-the-same-as-grok-debugger/175258 "2019-04-09T12:24:35Z")
hello guys, my grok debugger pattern output doesn't look like the logstash output in kibana!! any idea ! here is in kibana : --- ## [Logstash.filters.csv :exception=\>#\}](https://discuss.elastic.co/t/logstash-filters-csv-exception-runtimeerror-invalid-fieldreference/175791)
**Author:** [@Zeal0us](https://discuss.elastic.co/u/Zeal0us)\ **Replies:** 5\ **Last updated:** [April 9, 2019, 12:14pm UTC](https://discuss.elastic.co/t/logstash-filters-csv-exception-runtimeerror-invalid-fieldreference/175791 "2019-04-09T12:14:24Z")
Getting lines like this: \[2019-04-08T08:13:50,464\]\[WARN \]\[logstash.filters.csv \] Error parsing csv {:field=\>"message", :source=\>"", :exception=\>#\} when trying to parse so… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=567) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=569)