# How to create mass amounts of test data in elasticsearch / Kibana **URL:** \ **Category:** Elasticsearch\ **Created:** [March 14, 2023, 8:59am UTC](https://discuss.elastic.co/t/how-to-create-mass-amounts-of-test-data-in-elasticsearch-kibana/327637 "2023-03-14T08:59:33Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![shelby](https://avatars.discourse-cdn.com/v4/letter/s/aeb1de/32.png) [@shelby](https://discuss.elastic.co/u/shelby)\ **Post date:** [March 14, 2023, 8:59am UTC](https://discuss.elastic.co/t/how-to-create-mass-amounts-of-test-data-in-elasticsearch-kibana/327637/1 "2023-03-14T08:59:33Z")
0 I need to test some logic I have written in Kibana (Vega scripts). For this I require quite a bit of data to generate the graphs. I am currently doing this manually with statements such as: > post /metrics-hardware-xxx/\_doc {"@timestamp":"2023-03-11T15:00", "system\_Status": 1 } > post /metrics-hardware-xxx/\_doc {"@timestamp":"2023-03-11T15:01", "system\_Status": 1 } > post /metrics-hardware-xxx/\_doc {"@timestamp":"2023-03-11T15:02", "system\_Status": 0 } BUt I would like to have some kind of for loop where I can loop through timestamps 1 per minute over several days and assign a random system status either 0,1,2. Is this possible in Elastic? It is straightforward with a stored procedure in most sql DBs, but I am not sure where to start , and seems there is not much info on the web. ---
**Author:** ![carly.richmond](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carly.richmond/32/104935_2.png) [@carly.richmond](https://discuss.elastic.co/u/carly.richmond)\ **Post date:** [March 14, 2023, 9:34am UTC](https://discuss.elastic.co/t/how-to-create-mass-amounts-of-test-data-in-elasticsearch-kibana/327637/2 "2023-03-14T09:34:58Z")
Hi @shelby, There's not a capability to add a large volume of test data directly in Elasticsearch aside from the [`_bulk`](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html) upload endpoint with generated data. Since you want data in a specific format the demo sets available on the Kibana home page will not be what you're looking for. But there are several ways you could automate the ingestion of data volume using other sources. Have you considered either: 1. Writing a script in a programming language your comfortable with such as Python and then push the data using the [Elasticsearch client for that language](https://www.elastic.co/guide/en/elasticsearch/client/index.html)? 2. Using Logstash to ingest data from a file or DB (if you have another source) into Elasticsearch? 3. Making use of a community project such as [elasticsearch-sample-data-generator](https://github.com/ssi-anik/elasticsearch-sample-data-generator). Just a warning that project hasn't been updated since 2021 so you need to consider if the version of Elasticsearch you are running is compatible with that version. 4. Using a JSON generator and uploading the data, similar to the steps (in this post)[[How to create sample data to perform searches using elastic search? - DevOpsSchool.com](https://www.devopsschool.com/blog/how-to-create-sample-data-to-perform-searches-using-elastic-search/)] Do any of those options work for you? ---
**Author:** ![shelby](https://avatars.discourse-cdn.com/v4/letter/s/aeb1de/32.png) [@shelby](https://discuss.elastic.co/u/shelby)\ **Post date:** [March 14, 2023, 11:57am UTC](https://discuss.elastic.co/t/how-to-create-mass-amounts-of-test-data-in-elasticsearch-kibana/327637/3 "2023-03-14T11:57:21Z")
Hello, many thanks for hte follow-up, yes I am currently experimenting with \_bulk using curl. This is an excellent option for my case. ---
**Author:** ![shelby](https://avatars.discourse-cdn.com/v4/letter/s/aeb1de/32.png) [@shelby](https://discuss.elastic.co/u/shelby)\ **Post date:** [March 15, 2023, 7:40am UTC](https://discuss.elastic.co/t/how-to-create-mass-amounts-of-test-data-in-elasticsearch-kibana/327637/4 "2023-03-15T07:40:37Z")
Hello, I have some follow-up questions, I am still trying to get bulk insertion to work, and I am now testingthe elasticsearch c# client. I copied the very simple example in the doc: > var settings = new ElasticsearchClientSettings(new Uri(ELASTIC\_URI)) > .Authentication(new BasicAuthentication(USER, PWD)); > var client = new ElasticsearchClient(settings); > var response = await client.CountAsync(); > Console.WriteLine($"success={response.IsSuccess()} : {response.DebugInformation}"); However I am having connectivity issues. 1. whether or not I use he correct USER/PWD combination, I always get the same error message: Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on TransportConfiguration to force it to be set on the response.\> # Response: {"statusCode":404,"error":"Not Found","message":"Not Found"} 1. I am not sure what I should use for elastic\_URI, particularly for the PORT settings: I have [xxxx.yy.zz.aws.elastic-cloud.com:9243](http://xxxx.yy.zz.aws.elastic-cloud.com:9243) is the port setting correct? where can I find the URI? is it the same one that I use in the DEV Console elasticsearch? Do I need a "/" at the end? Most of the info on the webthat I found show examples where people are working with a local elastic installation and the port seems to always be 9200, so I am not sure of the url:port combination for a cloud setup. 1. I also spent quite a bit of time trying to connect using the certificate fingerprint which I generated with openssl according to the online documentation, I have similar issues there (I can try again and post more details on that as well). Any help is much appreciated, it is quite important or us to be able to manipulate the data remotely in bulk. ---
**Author:** ![carly.richmond](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carly.richmond/32/104935_2.png) [@carly.richmond](https://discuss.elastic.co/u/carly.richmond)\ **Post date:** [March 15, 2023, 11:11am UTC](https://discuss.elastic.co/t/how-to-create-mass-amounts-of-test-data-in-elasticsearch-kibana/327637/5 "2023-03-15T11:11:44Z")
Hi @shelby , Your URL should be a combination of your Elasticsearch URL and port separated by a colon. You can copy the base URL from the Cloud home screen as shown in the below screenshot: ![Screenshot 2023-03-15 at 07.08.18](https://us1.discourse-cdn.com/elastic/original/3X/f/1/f17d77d699c17625db157700bd717f23c452fbd8.png) Just like local installations, port 9200 should be used for API calls for a cloud deployment. So for example with copied URL `https://my-deployment.com`, you need to specify `https://my-deployment.com:9200`. Hope that helps! Do let me know how you get on. ---
**Author:** ![shelby](https://avatars.discourse-cdn.com/v4/letter/s/aeb1de/32.png) [@shelby](https://discuss.elastic.co/u/shelby)\ **Post date:** [March 16, 2023, 8:35am UTC](https://discuss.elastic.co/t/how-to-create-mass-amounts-of-test-data-in-elasticsearch-kibana/327637/6 "2023-03-16T08:35:49Z")
Hello, thanks again for your follow-up. Indeed, it seems the port and uri was the issue, however I was not able to get the endpoint to work. I was able to get the authentication working and I am now able to connect using the CLOUD\_ID and user/pwd. If anyone else has the issue here is my code: > ``` > var authentication = new BasicAuthentication(USER, PWD); > var client = new ElasticsearchClient(CLOUD_ID,authentication); > var response = await client.CountAsync(); > > ``` I will start experimentingwith the API which looks very powerful, I expect to easily be able to bulk insert in a flexible way. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 13, 2023, 8:36am UTC](https://discuss.elastic.co/t/how-to-create-mass-amounts-of-test-data-in-elasticsearch-kibana/327637/7 "2023-04-13T08:36:48Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.