# Elastic Stack **URL:** https://discuss.elastic.co/c/elastic-stack/81.md?page=1773 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1774 --- ## [Lumberjack error: Client write error, trying connect (..syswrite, write\_sync, multi\_receive, output\_batch, worker\_loop, start\_workers)](https://discuss.elastic.co/t/lumberjack-error-client-write-error-trying-connect-syswrite-write-sync-multi-receive-output-batch-worker-loop-start-workers/250557)
**Author:** [@Stanvv](https://discuss.elastic.co/u/Stanvv)\ **Replies:** 4\ **Last updated:** [October 2, 2020, 1:21pm UTC](https://discuss.elastic.co/t/lumberjack-error-client-write-error-trying-connect-syswrite-write-sync-multi-receive-output-batch-worker-loop-start-workers/250557 "2020-10-02T13:21:09Z")
When trying to forward data via the lumberjack protocol, we get the following error: See full error\[2020-09-30T20:23:04,014\]\[ERROR\]\[logstash.outputs.lumberjack\] Client write error, trying connect {:e=\>#\ **Author:** [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Replies:** 6\ **Last updated:** [October 2, 2020, 12:53pm UTC](https://discuss.elastic.co/t/elk-cluster-monitoring-with-metricbeat/249224 "2020-10-02T12:53:54Z") I am trying to set my cluster with metricbeat monitoring. follow the document. setup everything but all my node just keep changing what am i missing? elasticsearch.yml file on all system xpack.monitoring.collection.e… --- ## [Filter\_path in POST Body](https://discuss.elastic.co/t/filter-path-in-post-body/250776)
**Author:** [@Dev-Flo](https://discuss.elastic.co/u/Dev-Flo)\ **Replies:** 2\ **Last updated:** [October 2, 2020, 12:09pm UTC](https://discuss.elastic.co/t/filter-path-in-post-body/250776 "2020-10-02T12:09:57Z")
Hey, the responses I get from Elastisearch are rather huge. I've found out that it is possible to remove data of responses by using response filtering: Is there any way to put the filter inside a POST body so that I … --- ## [Static date conversion arbitrary failed](https://discuss.elastic.co/t/static-date-conversion-arbitrary-failed/250783)
**Author:** [@sesti](https://discuss.elastic.co/u/sesti)\ **Replies:** 0\ **Last updated:** [October 2, 2020, 12:09pm UTC](https://discuss.elastic.co/t/static-date-conversion-arbitrary-failed/250783 "2020-10-02T12:09:39Z")
For some reason I don't understand, logstash randomly fails to convert a date that doesn't change. date format: Fri Oct 2 error message: error type mapper\_parsiing\_exception ... with format \[strict\_date\_optionnal\_time\] … --- ## [Kibana is not able to fetch data from kibana collector](https://discuss.elastic.co/t/kibana-is-not-able-to-fetch-data-from-kibana-collector/250694)
**Author:** [@sudhakar1](https://discuss.elastic.co/u/sudhakar1)\ **Replies:** 7\ **Last updated:** [October 2, 2020, 11:55am UTC](https://discuss.elastic.co/t/kibana-is-not-able-to-fetch-data-from-kibana-collector/250694 "2020-10-02T11:55:39Z")
Kibana is not able to fetch data from kibana collector error is showing as screen shot below. Also Elastic Search is crashing , I am running ES 6.5.4 http://localhost:9200/\_cluster/stats {"\_nodes":{"total":1,"successfu… --- ## [Create visualization in Kibana](https://discuss.elastic.co/t/create-visualization-in-kibana/250109)
**Author:** [@malsioufi](https://discuss.elastic.co/u/malsioufi)\ **Replies:** 4\ **Last updated:** [October 2, 2020, 11:55am UTC](https://discuss.elastic.co/t/create-visualization-in-kibana/250109 "2020-10-02T11:55:27Z")
I have a csv file with data structure something like this I managed to ingest this data to elasticsearch via the import interface in Kibana. and now I have an index "example" with a related index pattern and the docum… --- ## [Test run fails](https://discuss.elastic.co/t/test-run-fails/250706)
**Author:** [@Craig2188](https://discuss.elastic.co/u/Craig2188)\ **Replies:** 7\ **Last updated:** [October 2, 2020, 10:43am UTC](https://discuss.elastic.co/t/test-run-fails/250706 "2020-10-02T10:43:56Z")
Hi, I have Logstash (on a seperate server to ES) set to run as a service, but wanted to test it first by running \\bin\\logstash.bat -f .\\config\\logstash-conf.conf However when I run this it appears to start but then st… --- ## [ES service keeps crashing](https://discuss.elastic.co/t/es-service-keeps-crashing/250661)
**Author:** [@Craig2188](https://discuss.elastic.co/u/Craig2188)\ **Replies:** 11\ **Last updated:** [October 2, 2020, 10:38am UTC](https://discuss.elastic.co/t/es-service-keeps-crashing/250661 "2020-10-02T10:38:48Z")
Hi, Just realised that after I start the service for Elastic it stops running after about 2 minutes. Checked the event logs and see some errors: Application: elasticsearch.exe Framework Version: v4.0.30319 Descripti… --- ## [Lostash and arcsight integration](https://discuss.elastic.co/t/lostash-and-arcsight-integration/247434)
**Author:** [@sidhesh\_kumar](https://discuss.elastic.co/u/sidhesh_kumar)\ **Replies:** 9\ **Last updated:** [October 2, 2020, 10:10am UTC](https://discuss.elastic.co/t/lostash-and-arcsight-integration/247434 "2020-10-02T10:10:50Z")
unable to configure smart connector on logstash content of logstash.yml modules: name: arcsight var.input.eventbroker.bootstrap\_servers: "XXXX.XXXXXX.XXXXX:1515" var.input.eventbroker.topics: "eb\_topic" var.elasti… --- ## [Filebeat 7.9.1 parsing http\_json data](https://discuss.elastic.co/t/filebeat-7-9-1-parsing-http-json-data/250314)
**Author:** [@cdroberts](https://discuss.elastic.co/u/cdroberts)\ **Replies:** 9\ **Last updated:** [October 2, 2020, 10:10am UTC](https://discuss.elastic.co/t/filebeat-7-9-1-parsing-http-json-data/250314 "2020-10-02T10:10:37Z")
Hi, I am having an issue parsing http\_json data with Filebeat. In the logs I can see the events come through fine. 2020-09-29T18:16:19.606+1000 DEBUG \[processors\] processing/processors.go:187 Publish event: { "@times… --- ## [Filter latest record for each user and aggregate on different field in elasticsearch](https://discuss.elastic.co/t/filter-latest-record-for-each-user-and-aggregate-on-different-field-in-elasticsearch/250435)
**Author:** [@matrix1](https://discuss.elastic.co/u/matrix1)\ **Replies:** 15\ **Last updated:** [October 2, 2020, 10:05am UTC](https://discuss.elastic.co/t/filter-latest-record-for-each-user-and-aggregate-on-different-field-in-elasticsearch/250435 "2020-10-02T10:05:18Z")
Need to filter the latest record for each user and aggregate the count on each vote type. Still cannot able to figure out. any help pls? {user:"sam", vote: 20, "createdDate": 1601285178238} {user:"sam", vote: 21, "crea… --- ## [Elasticsearch SQL retrieve more than 1000](https://discuss.elastic.co/t/elasticsearch-sql-retrieve-more-than-1000/250614)
**Author:** [@ananya\_ramesh](https://discuss.elastic.co/u/ananya_ramesh)\ **Replies:** 1\ **Last updated:** [October 2, 2020, 9:38am UTC](https://discuss.elastic.co/t/elasticsearch-sql-retrieve-more-than-1000/250614 "2020-10-02T09:38:02Z")
Implementing SQL query in Elasticsearch, not able to extract records more than 1000 in spite of LIMIT \> 1000. Tried "index.mapping.total\_fields.limit": 5000 , but did not help. POST \_sql?format=txt { "query": "SELECT… --- ## [How to parse functionbeat cloudwatch logs](https://discuss.elastic.co/t/how-to-parse-functionbeat-cloudwatch-logs/250765)
**Author:** [@Kay\_Khan](https://discuss.elastic.co/u/Kay_Khan)\ **Replies:** 0\ **Last updated:** [October 2, 2020, 8:57am UTC](https://discuss.elastic.co/t/how-to-parse-functionbeat-cloudwatch-logs/250765 "2020-10-02T08:57:34Z")
I have recently setup functionbeat to ship aws cloudwatch logs to elasticsearch. How do i format the message into multiple fields; timestamp, log\_id, log\_level and the actual message. I want to parse the message fields… --- ## [Ldap user authen sam account option](https://discuss.elastic.co/t/ldap-user-authen-sam-account-option/250767)
**Author:** [@lusynda](https://discuss.elastic.co/u/lusynda)\ **Replies:** 0\ **Last updated:** [October 2, 2020, 9:04am UTC](https://discuss.elastic.co/t/ldap-user-authen-sam-account-option/250767 "2020-10-02T09:04:54Z")
Hi all our organization have some specific rule to the naming of dn for example mine would me "cn=Something something,ou=test..." and my samaccount name would be soso@test.com and now when i have the role mapping to a… --- ## [Add Filter in discovery](https://discuss.elastic.co/t/add-filter-in-discovery/250333)
**Author:** [@graimato](https://discuss.elastic.co/u/graimato)\ **Replies:** 2\ **Last updated:** [October 2, 2020, 8:39am UTC](https://discuss.elastic.co/t/add-filter-in-discovery/250333 "2020-10-02T08:39:59Z")
Dear All, I'm trying to filter logs but when I choose value of the field the option menu does not show all possible values but only one part. there is a way to force it to show all possibilities? thanks for your time --- ## [How many shards for 5 millions of groups?](https://discuss.elastic.co/t/how-many-shards-for-5-millions-of-groups/250756)
**Author:** [@harisk](https://discuss.elastic.co/u/harisk)\ **Replies:** 3\ **Last updated:** [October 2, 2020, 8:26am UTC](https://discuss.elastic.co/t/how-many-shards-for-5-millions-of-groups/250756 "2020-10-02T08:26:28Z")
Now we have about 5 millions of groups and 100 millions of users. There are about 120 million messages every day users sent to the group, approximate 12GiB. We'd like to use Elasticsearch to search group messages with te… --- ## [Exporting data from field with HUGE content](https://discuss.elastic.co/t/exporting-data-from-field-with-huge-content/250760)
**Author:** [@prabhatdreamz](https://discuss.elastic.co/u/prabhatdreamz)\ **Replies:** 0\ **Last updated:** [October 2, 2020, 8:14am UTC](https://discuss.elastic.co/t/exporting-data-from-field-with-huge-content/250760 "2020-10-02T08:14:40Z")
I want to export my ES (v6.0.0) data to csv, for which I have the tool (es2csv). However, one of the fields (message) stores commit messages, and quite a lot of the entries are HUGE. The content can run to multiple pages… --- ## [Filebeat timestamp processsor doesn't parse microseconds part](https://discuss.elastic.co/t/filebeat-timestamp-processsor-doesnt-parse-microseconds-part/248591)
**Author:** [@6NMgfDwZ3](https://discuss.elastic.co/u/6NMgfDwZ3)\ **Replies:** 3\ **Last updated:** [October 2, 2020, 7:58am UTC](https://discuss.elastic.co/t/filebeat-timestamp-processsor-doesnt-parse-microseconds-part/248591 "2020-10-02T07:58:21Z")
Hi, this is the first part of my filebeat.xml filebeat.inputs: - type: tcp enabled: true host: "localhost:9000" processors: - decode\_json\_fields: fields: \["message"\] process\_array: true t… --- ## [Kibana Word Cloud on Text Field](https://discuss.elastic.co/t/kibana-word-cloud-on-text-field/250523)
**Author:** [@spyderman4g63](https://discuss.elastic.co/u/spyderman4g63)\ **Replies:** 10\ **Last updated:** [October 2, 2020, 7:21am UTC](https://discuss.elastic.co/t/kibana-word-cloud-on-text-field/250523 "2020-10-02T07:21:16Z")
I'm trying to create a word cloud on a text field called "message". This is my configuration for the field in elasticsearch: { "my\_index" : { "mappings" : { "message" : { "full\_name" : "message", … --- ## [Send k8s logs with Filebeat to separate indices](https://discuss.elastic.co/t/send-k8s-logs-with-filebeat-to-separate-indices/250632)
**Author:** [@karol\_k](https://discuss.elastic.co/u/karol_k)\ **Replies:** 1\ **Last updated:** [October 2, 2020, 7:02am UTC](https://discuss.elastic.co/t/send-k8s-logs-with-filebeat-to-separate-indices/250632 "2020-10-02T07:02:02Z")
Hi :slight\_smile: Currently I'm using self-hosted elastic stack but evaluating elastic cloud. I collect all the logs from multiple k8s clusters in a standard way with filebeat kubernetes module. There are application an… --- ## [How to process again documents already ingested](https://discuss.elastic.co/t/how-to-process-again-documents-already-ingested/250716)
**Author:** [@rschirin](https://discuss.elastic.co/u/rschirin)\ **Replies:** 3\ **Last updated:** [October 2, 2020, 6:59am UTC](https://discuss.elastic.co/t/how-to-process-again-documents-already-ingested/250716 "2020-10-02T06:59:53Z")
Hi all, I would like to know if there is a way to process again documents already saved into the indices. I noticed that my Logstash pipeline was not completed so now I know like to process a specific field. in my doc… --- ## [How to convert the date format - From DOUBLE to Datetime in KIbana elastic search](https://discuss.elastic.co/t/how-to-convert-the-date-format-from-double-to-datetime-in-kibana-elastic-search/250713)
**Author:** [@Prabhu.292](https://discuss.elastic.co/u/Prabhu.292)\ **Replies:** 2\ **Last updated:** [October 2, 2020, 6:09am UTC](https://discuss.elastic.co/t/how-to-convert-the-date-format-from-double-to-datetime-in-kibana-elastic-search/250713 "2020-10-02T06:09:53Z")
Could you please help with the below issue. Is there anyway can we change the date format for the below mentioned columns? Time\_token : 1.6013861887237842E9 Created\_at: 1.600254825E9 I want to create the dashboards b… --- ## [Trouble setting up TLS in Kibana with Docker](https://discuss.elastic.co/t/trouble-setting-up-tls-in-kibana-with-docker/250041)
**Author:** [@Cyberdemic](https://discuss.elastic.co/u/Cyberdemic)\ **Replies:** 15\ **Last updated:** [October 2, 2020, 5:20am UTC](https://discuss.elastic.co/t/trouble-setting-up-tls-in-kibana-with-docker/250041 "2020-10-02T05:20:49Z")
I'm trying to setup ELK stack in Docker Swarm following this guide with some modifications. I've tried a number of different options, and while it's helped me learn more of ELK, I still haven't managed to get it working… --- ## [Elasticsearch/Kibana 7.9 - Troubleshooting](https://discuss.elastic.co/t/elasticsearch-kibana-7-9-troubleshooting/250735)
**Author:** [@kercio](https://discuss.elastic.co/u/kercio)\ **Replies:** 3\ **Last updated:** [October 2, 2020, 2:32am UTC](https://discuss.elastic.co/t/elasticsearch-kibana-7-9-troubleshooting/250735 "2020-10-02T02:32:43Z")
What is the best way to troubleshoot in Elasticsearch/Kibana to find the root cause why only part of my raw data is getting indexed? Currently I have Kibana 6.2 and 7.9 in my environment. While in Kibana 6.2 one of the… --- ## [Find number of matching pair, display in pie chart](https://discuss.elastic.co/t/find-number-of-matching-pair-display-in-pie-chart/249503)
**Author:** [@nameisnotimportant](https://discuss.elastic.co/u/nameisnotimportant)\ **Replies:** 5\ **Last updated:** [October 2, 2020, 1:18am UTC](https://discuss.elastic.co/t/find-number-of-matching-pair-display-in-pie-chart/249503 "2020-10-02T01:18:05Z")
Hi, Say i have a table of records like below: Is it possible to use Kibana visualization to find and display the total number of matching pair where FundA is follow by DD (within the same SessionID) and the total num… --- ## [Kibana canvasrender options](https://discuss.elastic.co/t/kibana-canvasrender-options/250558)
**Author:** [@Robin020](https://discuss.elastic.co/u/Robin020)\ **Replies:** 4\ **Last updated:** [October 1, 2020, 11:14pm UTC](https://discuss.elastic.co/t/kibana-canvasrender-options/250558 "2020-10-01T23:14:52Z")
Hello, Is there documentation about canvasrender options? I am trying to change the filling color of a gauge in canvas. --- ## [Split objects in a nested array into separate record](https://discuss.elastic.co/t/split-objects-in-a-nested-array-into-separate-record/250425)
**Author:** [@jasoninmel](https://discuss.elastic.co/u/jasoninmel)\ **Replies:** 12\ **Last updated:** [October 1, 2020, 11:09pm UTC](https://discuss.elastic.co/t/split-objects-in-a-nested-array-into-separate-record/250425 "2020-10-01T23:09:49Z")
Hi there, I have the following data structure in Elasticsearch. { "id": "559466", "name": "test", "availability": \[ { "assetId": "559466", "type": "pen", "complete": true, "availableFr… --- ## [Logstash if condition](https://discuss.elastic.co/t/logstash-if-condition/250689)
**Author:** [@Yotam\_Mazurik](https://discuss.elastic.co/u/Yotam_Mazurik)\ **Replies:** 4\ **Last updated:** [October 1, 2020, 10:04pm UTC](https://discuss.elastic.co/t/logstash-if-condition/250689 "2020-10-01T22:04:45Z")
Hey, i'm new to logstash and got stuck while trying to create an if condition... The structure of my if condition: if (\[process\]\[executable\] not in \[processes\]\["%{\[process\]\[hash\]\[md5\]}"\]\[paths\]) { .... } w… --- ## [Dissect failure for no obvious reason](https://discuss.elastic.co/t/dissect-failure-for-no-obvious-reason/250634)
**Author:** [@Haptomai](https://discuss.elastic.co/u/Haptomai)\ **Replies:** 3\ **Last updated:** [October 1, 2020, 9:47pm UTC](https://discuss.elastic.co/t/dissect-failure-for-no-obvious-reason/250634 "2020-10-01T21:47:49Z")
Hi there, For no obvious (to me) reason, a certain message coming from filebeat that always fell in error on logstash processing. Here is the sample: Oct 1 09:38:12 ff434-srv1 MC: debug {74308} \[ApiUiJob\] PlanJob()… --- ## [504 Error due to large number of routing parameters](https://discuss.elastic.co/t/504-error-due-to-large-number-of-routing-parameters/250742)
**Author:** [@prosens](https://discuss.elastic.co/u/prosens)\ **Replies:** 2\ **Last updated:** [October 1, 2020, 9:42pm UTC](https://discuss.elastic.co/t/504-error-due-to-large-number-of-routing-parameters/250742 "2020-10-01T21:42:50Z")
We have a large number of shards. The documents are routed using user ids. There are use cases where we need to search across multiple users. When we specify a large number of user ids as the routing parameter, we see a … [Previous page](https://discuss.elastic.co/c/elastic-stack/81.md?page=1772) [Next page](https://discuss.elastic.co/c/elastic-stack/81.md?page=1774)