Only one record gets created in the Elastic search

Elastic Stack Logstash
1

I am using Logstash and created a conf file which allows me to input data into Elasticsearch. The data is related to git statistics from azure as I am creating a dashboard which will help to see information such as total commits, total PRs by an author etc. I am using http poller plugin for the same. Here's my conf file:

input {
    http_poller {
      urls => {
        restapi => {
            method => get
            url => "https://dev.azure.com/Org/Repo/_apis/git/pullrequests?$top=500&searchCriteria.status=completed&api-version=7.0"
            headers => {
                Accept =>  "application/json"
            }
            auth => {
             user => "****"
             password => "***"
            }
        }
      }
      codec => "json"
      schedule => {
        every => "1800m"
      }
    }
}

filter {
        json{
            source => "message"
        }
}

output {
  elasticsearch {
    hosts => ["https://127.0.0.1:9200"]
    index => "pollertwoindex"
    user => "elastic"
    password => "p6****"
    ssl_certificate_verification => false
  }
  stdout{
   codec => rubydebug
  }
}

Data is successfully imported to Elasticsearch as I can see on Kibana and an index is also created as I specified in the conf file.
But the problem is there's only one record which is being created.

I tried to import the same data before through .csv file and configuring logstash for the same. It creates as many records as there are in the .csv file but not happening with the http poller when I am fetching data through REST API.

I am new to Elastic Search. Can anybody help?

2

I am still following this. Please help

3

Hello,

Please share the document you have in elasticsearch, how it does look like? Go into Kibana Discover, expand the document, click on the JSON tab and copy and share the document.

4

The document is very large but a part of it looks like this

{
  "_index": "pollertwoindex",
  "_id": "E9NZdIoBo4nWxnmf_o-O",
  "_version": 1,
  "_score": 1,
  "_ignored": [
    "value.description.keyword",
    "event.original.keyword",
    "value.completionOptions.mergeCommitMessage.keyword"
  ],
  "_source": {
    "@version": "1",
    "@timestamp": "2023-09-08T10:32:48.353655Z",
    "value": [
      {
        "closedDate": "2023-09-08T10:29:56.2036954Z",
        "lastMergeCommit": {
          "url": “xyz.com”,
          "commitId": "551267949b4defa08a68f6fb1f3bffaf0d42d47d"
        },
        "mergeId": "bdd5bc33-e90e-4344-a40c-16d3ba076d6a",
        "pullRequestId": 104344,
        "targetRefName": "refs/heads/feature/forms/web_master/PREPROD_cont-sites",
        "completionQueueTime": "2023-09-08T10:29:55.2945776Z",
        "codeReviewId": 104344,
        "mergeStatus": "succeeded",
        "createdBy": {
          "imageUrl":  “xyz.com”,
          "_links": {
            "avatar": {
              “xyz.com”
            }
          },
          "url":  “xyz.com”,
          "uniqueName":  “xyz.com”,
          "id": "04b364d7-21af-6609-8dbd-bbc584e3c0db",
          "descriptor": "aad.MDRiMzY0ZDctMjFhZi03NjA5LThkYmQtYmJjNTg0ZTNjMGRi"
        },
        "description": “dfgsdfgs”,
        "sourceRefName": "refs/heads/feature/forms/web_master/PREPROD_cont-sites_HL_0208",
        "supportsIterations": true,
        "title": “random fix,
        "reviewers": [
          {
            "displayName": “John”,
            "isFlagged": false,
            "uniqueName":  “xyz.com”,
            "vote": 10,
            "id": "1fc73d1a-87e0-6d93-93ba-4e2da45aa8b3",
            "imageUrl":  “xyz.com”,
            "_links": {
              "avatar": {
                "href":  “xyz.com”
5

From what you share I would assume that you have a large document where you have multiple items inside the value field, is that right?

This is pretty common as it is standard for APIs to return an array with the individual events.

In your case your response may be something like this:

{
    "value": [
        {"json of commit 1"},
        {"json of commit 2"},
        {"json of commit 3"},
        {"json of commit N"}
    ]
}

You can solve this and have one document per json using the split filter.

Change your fliter block to this.

filter {
    json{
        source => "message"
    }
    split {
        field => "value"
    }
}
1 Like
6

Thanks Leandro for the reply.
I changed my conf file. But now I am getting

java.lang.OutOfMemoryError: Java heap space

Pretty sure this is a common issue because large number of rows are getting created. How to increase the size or any other suggested solution?

7

Where did you get this? In Logstash logs?

What is the configuration of your jvm.options for Logstash? You will need to increase it.

8

Thanks Leandro!
I have set the jvm.options. But now Kibana shows me this error in the document discover:

ReleasableBytesStreamOutput cannot hold more than 2GB of data

To fix this I tried:

PUT _cluster/settings
{
  "persistent" : {
    "search.max_async_search_response_size" : "8000mb"
  }
}

But still getting the same error

9

Are you running Kibana and Logstash on the same machine?

What is the specification of the machine?

Do you still have just one document?

1 Like
10

Yes I'm running Kibana and Logstash on the same machine. I'm using a Mac with 16GB RAM. The earlier problem regd one document is solved now but I can't load the documents due to the above error.

11

Are you running Elasticsearch on the same machine as well?

How are you running it? Docker or directly on the OS? Have you changed the memory configuration for Elasticsearch?

Also, what value did you use in jvm.options for Logstash?

If you didn't change the memory settings for Elasticsearch it will per default use half of the memory you have, which will be 8 GB, then depending on how much memory you gave to Logstash your system may not have enough memory for Kibana for example.

I would also suggest that you reset this config to the default value until you find what is the root cause of your issue.

PUT _cluster/settings
{
  "persistent" : {
    "search.max_async_search_response_size" : null
  }
}
12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.