# \#none **URL:** https://discuss.elastic.co/tag/none.md?no_tags=true&page=1212 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1213 --- ## [Creating a counter out of log occurrences and scraping the log itself](https://discuss.elastic.co/t/creating-a-counter-out-of-log-occurrences-and-scraping-the-log-itself/250778)
**Author:** [@Jarek1](https://discuss.elastic.co/u/Jarek1)\ **Replies:** 1\ **Last updated:** [October 2, 2020, 1:38pm UTC](https://discuss.elastic.co/t/creating-a-counter-out-of-log-occurrences-and-scraping-the-log-itself/250778 "2020-10-02T13:38:26Z")
Hi Community, can you please advice me how to proceed with the following topic? I am feeding unstructured logs from 3rd party application to the logstash via UDP feed. Logs are categorized by using grok script. For som… --- ## [Overriding logstash.yml in pipelines.yml](https://discuss.elastic.co/t/overriding-logstash-yml-in-pipelines-yml/250757)
**Author:** [@dawiro](https://discuss.elastic.co/u/dawiro)\ **Replies:** 1\ **Last updated:** [October 2, 2020, 1:36pm UTC](https://discuss.elastic.co/t/overriding-logstash-yml-in-pipelines-yml/250757 "2020-10-02T13:36:49Z")
Hi, I am testing a pipelined configuration using the output isolator pattern. While this I am still using pipeline batch/worker settings from logstash.yml and would like to use pipeline.ordered for one of the outputs. W… --- ## [Can not parse valid JSON message in LogStash](https://discuss.elastic.co/t/can-not-parse-valid-json-message-in-logstash/250791)
**Author:** [@dencowboy](https://discuss.elastic.co/u/dencowboy)\ **Replies:** 0\ **Last updated:** [October 2, 2020, 1:27pm UTC](https://discuss.elastic.co/t/can-not-parse-valid-json-message-in-logstash/250791 "2020-10-02T13:27:00Z")
I'm using ElasticSearch 7.7 and logstash 7.9.2. This is my configuration: input { kinesis { kinesis\_stream\_name =\> "xxx-${ENV}" application\_name =\> "xxx-${ENV}" region =\> "eu-west-1" tags =\> \["log"\] … --- ## [Lumberjack error: Client write error, trying connect (..syswrite, write\_sync, multi\_receive, output\_batch, worker\_loop, start\_workers)](https://discuss.elastic.co/t/lumberjack-error-client-write-error-trying-connect-syswrite-write-sync-multi-receive-output-batch-worker-loop-start-workers/250557)
**Author:** [@Stanvv](https://discuss.elastic.co/u/Stanvv)\ **Replies:** 4\ **Last updated:** [October 2, 2020, 1:21pm UTC](https://discuss.elastic.co/t/lumberjack-error-client-write-error-trying-connect-syswrite-write-sync-multi-receive-output-batch-worker-loop-start-workers/250557 "2020-10-02T13:21:09Z")
When trying to forward data via the lumberjack protocol, we get the following error: See full error\[2020-09-30T20:23:04,014\]\[ERROR\]\[logstash.outputs.lumberjack\] Client write error, trying connect {:e=\>#\ **Author:** [@Dev-Flo](https://discuss.elastic.co/u/Dev-Flo)\ **Replies:** 2\ **Last updated:** [October 2, 2020, 12:09pm UTC](https://discuss.elastic.co/t/filter-path-in-post-body/250776 "2020-10-02T12:09:57Z") Hey, the responses I get from Elastisearch are rather huge. I've found out that it is possible to remove data of responses by using response filtering: Is there any way to put the filter inside a POST body so that I … --- ## [Static date conversion arbitrary failed](https://discuss.elastic.co/t/static-date-conversion-arbitrary-failed/250783)
**Author:** [@sesti](https://discuss.elastic.co/u/sesti)\ **Replies:** 0\ **Last updated:** [October 2, 2020, 12:09pm UTC](https://discuss.elastic.co/t/static-date-conversion-arbitrary-failed/250783 "2020-10-02T12:09:39Z")
For some reason I don't understand, logstash randomly fails to convert a date that doesn't change. date format: Fri Oct 2 error message: error type mapper\_parsiing\_exception ... with format \[strict\_date\_optionnal\_time\] … --- ## [Kibana is not able to fetch data from kibana collector](https://discuss.elastic.co/t/kibana-is-not-able-to-fetch-data-from-kibana-collector/250694)
**Author:** [@sudhakar1](https://discuss.elastic.co/u/sudhakar1)\ **Replies:** 7\ **Last updated:** [October 2, 2020, 11:55am UTC](https://discuss.elastic.co/t/kibana-is-not-able-to-fetch-data-from-kibana-collector/250694 "2020-10-02T11:55:39Z")
Kibana is not able to fetch data from kibana collector error is showing as screen shot below. Also Elastic Search is crashing , I am running ES 6.5.4 http://localhost:9200/\_cluster/stats {"\_nodes":{"total":1,"successfu… --- ## [Create visualization in Kibana](https://discuss.elastic.co/t/create-visualization-in-kibana/250109)
**Author:** [@malsioufi](https://discuss.elastic.co/u/malsioufi)\ **Replies:** 4\ **Last updated:** [October 2, 2020, 11:55am UTC](https://discuss.elastic.co/t/create-visualization-in-kibana/250109 "2020-10-02T11:55:27Z")
I have a csv file with data structure something like this I managed to ingest this data to elasticsearch via the import interface in Kibana. and now I have an index "example" with a related index pattern and the docum… --- ## [Test run fails](https://discuss.elastic.co/t/test-run-fails/250706)
**Author:** [@Craig2188](https://discuss.elastic.co/u/Craig2188)\ **Replies:** 7\ **Last updated:** [October 2, 2020, 10:43am UTC](https://discuss.elastic.co/t/test-run-fails/250706 "2020-10-02T10:43:56Z")
Hi, I have Logstash (on a seperate server to ES) set to run as a service, but wanted to test it first by running \\bin\\logstash.bat -f .\\config\\logstash-conf.conf However when I run this it appears to start but then st… --- ## [ES service keeps crashing](https://discuss.elastic.co/t/es-service-keeps-crashing/250661)
**Author:** [@Craig2188](https://discuss.elastic.co/u/Craig2188)\ **Replies:** 11\ **Last updated:** [October 2, 2020, 10:38am UTC](https://discuss.elastic.co/t/es-service-keeps-crashing/250661 "2020-10-02T10:38:48Z")
Hi, Just realised that after I start the service for Elastic it stops running after about 2 minutes. Checked the event logs and see some errors: Application: elasticsearch.exe Framework Version: v4.0.30319 Descripti… --- ## [Lostash and arcsight integration](https://discuss.elastic.co/t/lostash-and-arcsight-integration/247434)
**Author:** [@sidhesh\_kumar](https://discuss.elastic.co/u/sidhesh_kumar)\ **Replies:** 9\ **Last updated:** [October 2, 2020, 10:10am UTC](https://discuss.elastic.co/t/lostash-and-arcsight-integration/247434 "2020-10-02T10:10:50Z")
unable to configure smart connector on logstash content of logstash.yml modules: name: arcsight var.input.eventbroker.bootstrap\_servers: "XXXX.XXXXXX.XXXXX:1515" var.input.eventbroker.topics: "eb\_topic" var.elasti… --- ## [Why PVCs are being deleted and how to create role mappings with OIDC](https://discuss.elastic.co/t/why-pvcs-are-being-deleted-and-how-to-create-role-mappings-with-oidc/250053)
**Author:** [@cccs-eric](https://discuss.elastic.co/u/cccs-eric)\ **Replies:** 1\ **Last updated:** [October 2, 2020, 10:10am UTC](https://discuss.elastic.co/t/why-pvcs-are-being-deleted-and-how-to-create-role-mappings-with-oidc/250053 "2020-10-02T10:10:22Z")
Hi, I am setting up a new ECK instance using the Operator and I have two questions that I can’t find an answer for. 1- on this documentation page https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-volume-claim-te… --- ## [Filter latest record for each user and aggregate on different field in elasticsearch](https://discuss.elastic.co/t/filter-latest-record-for-each-user-and-aggregate-on-different-field-in-elasticsearch/250435)
**Author:** [@matrix1](https://discuss.elastic.co/u/matrix1)\ **Replies:** 15\ **Last updated:** [October 2, 2020, 10:05am UTC](https://discuss.elastic.co/t/filter-latest-record-for-each-user-and-aggregate-on-different-field-in-elasticsearch/250435 "2020-10-02T10:05:18Z")
Need to filter the latest record for each user and aggregate the count on each vote type. Still cannot able to figure out. any help pls? {user:"sam", vote: 20, "createdDate": 1601285178238} {user:"sam", vote: 21, "crea… --- ## [Ldap user authen sam account option](https://discuss.elastic.co/t/ldap-user-authen-sam-account-option/250767)
**Author:** [@lusynda](https://discuss.elastic.co/u/lusynda)\ **Replies:** 0\ **Last updated:** [October 2, 2020, 9:04am UTC](https://discuss.elastic.co/t/ldap-user-authen-sam-account-option/250767 "2020-10-02T09:04:54Z")
Hi all our organization have some specific rule to the naming of dn for example mine would me "cn=Something something,ou=test..." and my samaccount name would be soso@test.com and now when i have the role mapping to a… --- ## [Add Filter in discovery](https://discuss.elastic.co/t/add-filter-in-discovery/250333)
**Author:** [@graimato](https://discuss.elastic.co/u/graimato)\ **Replies:** 2\ **Last updated:** [October 2, 2020, 8:39am UTC](https://discuss.elastic.co/t/add-filter-in-discovery/250333 "2020-10-02T08:39:59Z")
Dear All, I'm trying to filter logs but when I choose value of the field the option menu does not show all possible values but only one part. there is a way to force it to show all possibilities? thanks for your time --- ## [How many shards for 5 millions of groups?](https://discuss.elastic.co/t/how-many-shards-for-5-millions-of-groups/250756)
**Author:** [@harisk](https://discuss.elastic.co/u/harisk)\ **Replies:** 3\ **Last updated:** [October 2, 2020, 8:26am UTC](https://discuss.elastic.co/t/how-many-shards-for-5-millions-of-groups/250756 "2020-10-02T08:26:28Z")
Now we have about 5 millions of groups and 100 millions of users. There are about 120 million messages every day users sent to the group, approximate 12GiB. We'd like to use Elasticsearch to search group messages with te… --- ## [Exporting data from field with HUGE content](https://discuss.elastic.co/t/exporting-data-from-field-with-huge-content/250760)
**Author:** [@prabhatdreamz](https://discuss.elastic.co/u/prabhatdreamz)\ **Replies:** 0\ **Last updated:** [October 2, 2020, 8:14am UTC](https://discuss.elastic.co/t/exporting-data-from-field-with-huge-content/250760 "2020-10-02T08:14:40Z")
I want to export my ES (v6.0.0) data to csv, for which I have the tool (es2csv). However, one of the fields (message) stores commit messages, and quite a lot of the entries are HUGE. The content can run to multiple pages… --- ## [Kibana Word Cloud on Text Field](https://discuss.elastic.co/t/kibana-word-cloud-on-text-field/250523)
**Author:** [@spyderman4g63](https://discuss.elastic.co/u/spyderman4g63)\ **Replies:** 10\ **Last updated:** [October 2, 2020, 7:21am UTC](https://discuss.elastic.co/t/kibana-word-cloud-on-text-field/250523 "2020-10-02T07:21:16Z")
I'm trying to create a word cloud on a text field called "message". This is my configuration for the field in elasticsearch: { "my\_index" : { "mappings" : { "message" : { "full\_name" : "message", … --- ## [How to process again documents already ingested](https://discuss.elastic.co/t/how-to-process-again-documents-already-ingested/250716)
**Author:** [@rschirin](https://discuss.elastic.co/u/rschirin)\ **Replies:** 3\ **Last updated:** [October 2, 2020, 6:59am UTC](https://discuss.elastic.co/t/how-to-process-again-documents-already-ingested/250716 "2020-10-02T06:59:53Z")
Hi all, I would like to know if there is a way to process again documents already saved into the indices. I noticed that my Logstash pipeline was not completed so now I know like to process a specific field. in my doc… --- ## [Пустые поля](https://discuss.elastic.co/t/topic/250335)
**Author:** [@artem33region](https://discuss.elastic.co/u/artem33region)\ **Replies:** 6\ **Last updated:** [October 2, 2020, 6:44am UTC](https://discuss.elastic.co/t/topic/250335 "2020-10-02T06:44:12Z")
Всем добрый день, есть пустые поля, которые нужно заполнить например нулями, вот только как сделать условие проверки на пустое поле? Поля не все пустые, тип поля number. Пробовал разные проверки - ничего не выходит... … --- ## [How to convert the date format - From DOUBLE to Datetime in KIbana elastic search](https://discuss.elastic.co/t/how-to-convert-the-date-format-from-double-to-datetime-in-kibana-elastic-search/250713)
**Author:** [@Prabhu.292](https://discuss.elastic.co/u/Prabhu.292)\ **Replies:** 2\ **Last updated:** [October 2, 2020, 6:09am UTC](https://discuss.elastic.co/t/how-to-convert-the-date-format-from-double-to-datetime-in-kibana-elastic-search/250713 "2020-10-02T06:09:53Z")
Could you please help with the below issue. Is there anyway can we change the date format for the below mentioned columns? Time\_token : 1.6013861887237842E9 Created\_at: 1.600254825E9 I want to create the dashboards b… --- ## [Elasticsearch/Kibana 7.9 - Troubleshooting](https://discuss.elastic.co/t/elasticsearch-kibana-7-9-troubleshooting/250735)
**Author:** [@kercio](https://discuss.elastic.co/u/kercio)\ **Replies:** 3\ **Last updated:** [October 2, 2020, 2:32am UTC](https://discuss.elastic.co/t/elasticsearch-kibana-7-9-troubleshooting/250735 "2020-10-02T02:32:43Z")
What is the best way to troubleshoot in Elasticsearch/Kibana to find the root cause why only part of my raw data is getting indexed? Currently I have Kibana 6.2 and 7.9 in my environment. While in Kibana 6.2 one of the… --- ## [Find number of matching pair, display in pie chart](https://discuss.elastic.co/t/find-number-of-matching-pair-display-in-pie-chart/249503)
**Author:** [@nameisnotimportant](https://discuss.elastic.co/u/nameisnotimportant)\ **Replies:** 5\ **Last updated:** [October 2, 2020, 1:18am UTC](https://discuss.elastic.co/t/find-number-of-matching-pair-display-in-pie-chart/249503 "2020-10-02T01:18:05Z")
Hi, Say i have a table of records like below: Is it possible to use Kibana visualization to find and display the total number of matching pair where FundA is follow by DD (within the same SessionID) and the total num… --- ## [SIEM feature request](https://discuss.elastic.co/t/siem-feature-request/250340)
**Author:** [@lusynda](https://discuss.elastic.co/u/lusynda)\ **Replies:** 4\ **Last updated:** [October 1, 2020, 11:22pm UTC](https://discuss.elastic.co/t/siem-feature-request/250340 "2020-10-01T23:22:36Z")
Hi, As i was scrolling down the huge pile of log and then i come up with this ideal for SIEM that i really want What i want is the ability to detect multiple event subsequently, what i mean is that for example i have a… --- ## [Kibana canvasrender options](https://discuss.elastic.co/t/kibana-canvasrender-options/250558)
**Author:** [@Robin020](https://discuss.elastic.co/u/Robin020)\ **Replies:** 4\ **Last updated:** [October 1, 2020, 11:14pm UTC](https://discuss.elastic.co/t/kibana-canvasrender-options/250558 "2020-10-01T23:14:52Z")
Hello, Is there documentation about canvasrender options? I am trying to change the filling color of a gauge in canvas. --- ## [Split objects in a nested array into separate record](https://discuss.elastic.co/t/split-objects-in-a-nested-array-into-separate-record/250425)
**Author:** [@jasoninmel](https://discuss.elastic.co/u/jasoninmel)\ **Replies:** 12\ **Last updated:** [October 1, 2020, 11:09pm UTC](https://discuss.elastic.co/t/split-objects-in-a-nested-array-into-separate-record/250425 "2020-10-01T23:09:49Z")
Hi there, I have the following data structure in Elasticsearch. { "id": "559466", "name": "test", "availability": \[ { "assetId": "559466", "type": "pen", "complete": true, "availableFr… --- ## [Logstash if condition](https://discuss.elastic.co/t/logstash-if-condition/250689)
**Author:** [@Yotam\_Mazurik](https://discuss.elastic.co/u/Yotam_Mazurik)\ **Replies:** 4\ **Last updated:** [October 1, 2020, 10:04pm UTC](https://discuss.elastic.co/t/logstash-if-condition/250689 "2020-10-01T22:04:45Z")
Hey, i'm new to logstash and got stuck while trying to create an if condition... The structure of my if condition: if (\[process\]\[executable\] not in \[processes\]\["%{\[process\]\[hash\]\[md5\]}"\]\[paths\]) { .... } w… --- ## [Dissect failure for no obvious reason](https://discuss.elastic.co/t/dissect-failure-for-no-obvious-reason/250634)
**Author:** [@Haptomai](https://discuss.elastic.co/u/Haptomai)\ **Replies:** 3\ **Last updated:** [October 1, 2020, 9:47pm UTC](https://discuss.elastic.co/t/dissect-failure-for-no-obvious-reason/250634 "2020-10-01T21:47:49Z")
Hi there, For no obvious (to me) reason, a certain message coming from filebeat that always fell in error on logstash processing. Here is the sample: Oct 1 09:38:12 ff434-srv1 MC: debug {74308} \[ApiUiJob\] PlanJob()… --- ## [504 Error due to large number of routing parameters](https://discuss.elastic.co/t/504-error-due-to-large-number-of-routing-parameters/250742)
**Author:** [@prosens](https://discuss.elastic.co/u/prosens)\ **Replies:** 2\ **Last updated:** [October 1, 2020, 9:42pm UTC](https://discuss.elastic.co/t/504-error-due-to-large-number-of-routing-parameters/250742 "2020-10-01T21:42:50Z")
We have a large number of shards. The documents are routed using user ids. There are use cases where we need to search across multiple users. When we specify a large number of user ids as the routing parameter, we see a … --- ## [Elasticdump users: does anyone else worry about WARNings on dependencies?](https://discuss.elastic.co/t/elasticdump-users-does-anyone-else-worry-about-warnings-on-dependencies/250480)
**Author:** [@viu](https://discuss.elastic.co/u/viu)\ **Replies:** 2\ **Last updated:** [October 1, 2020, 8:27pm UTC](https://discuss.elastic.co/t/elasticdump-users-does-anyone-else-worry-about-warnings-on-dependencies/250480 "2020-10-01T20:27:57Z")
Hi there, I am a newbie in this community, but lurked as a visitor for some time. I work with a very security aware IT/server administration, and it has made me semi-paranoid. The Elastick stack in use is an older ver… [Previous page](https://discuss.elastic.co/tag/none.md?no_tags=true&page=1211) [Next page](https://discuss.elastic.co/tag/none.md?no_tags=true&page=1213)