# Another Cloudflare integration issue

**URL:** <https://discuss.elastic.co/t/another-cloudflare-integration-issue/320085>\
**Category:** Elastic Agent\
**Tags:** integrations\
**Created:** [November 29, 2022, 7:09pm UTC](https://discuss.elastic.co/t/another-cloudflare-integration-issue/320085 "2022-11-29T19:09:26Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![ethhack](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethhack/32/109565_2.png) [@ethhack](https://discuss.elastic.co/u/ethhack)\
**Post date:** [November 29, 2022, 7:09pm UTC](https://discuss.elastic.co/t/another-cloudflare-integration-issue/320085/1 "2022-11-29T19:09:26Z")

</div>

Similar to [this post](https://discuss.elastic.co/t/cloudflare-integration-not-working/312631) for which I see no resolution anywhere, I have an issue with the Cloudflare logpull integration erroring out about the 168h maximum:

"message":"Error while processing http request: failed to execute rf.collectResponse: failed to execute http client.Do: server responded with status code 400: bad query: error parsing time: invalid time range: too early: logs older than 168h0m0s are not available"

All settings within the integration are at or below the 24h setting, even. I see nowhere that I can adjust a time setting for the period for which to gather that is having ANY effect on this error, and it appears that no logs are coming in. A manual curl against the API, with a date range of 1 hour is having no issues, so I'd assume it's something either hard-coded or 'stuck' from a previous config. This was working until a few weeks ago, with no issue, but now I can't seem to work around it.

Our policy (with relevant ID's and Key's edited out):

```auto
{
  "item": {
    "id": "f5e823df-87d9-498f-bc49-3bc8fb08b5b4",
    "version": "WzY4MDg5NDQsNF0=",
    "name": "cloudflare-1",
    "namespace": "default",
    "description": "",
    "package": {
      "name": "cloudflare",
      "title": "Cloudflare",
      "version": "2.3.0"
    },
    "enabled": true,
    "policy_id": "142374b0-368d-11ed-8fc9-9b05019573b8",
    "inputs": [
      {
        "policy_template": "cloudflare",
        "streams": [
          {
            "compiled_stream": {
              "cursor": {
                "last_timestamp": {
                  "fail_on_template_error": true,
                  "value": "[[.first_event.when]]"
                }
              },
              "response.pagination": [
                {
                  "set": {
                    "fail_on_template_error": true,
                    "value": "[[if (ne (len .last_response.body.result) 0)]][[add .last_response.page 1]][[end]]",
                    "target": "url.params.page"
                  }
                }
              ],
              "response.split": {
                "target": "body.result"
              },
              "request.method": "GET",
              "interval": "1h",
              "request.url": "https://api.cloudflare.com/client/v4/accounts/<our account id>/audit_logs?page=1&direction=desc",
              "processors": [
                {
                  "add_fields": {
                    "fields": {
                      "account_id": "<our account id>"
                    },
                    "target": "_config"
                  }
                }
              ],
              "config_version": "2",
              "request.timeout": "60s",
              "request.transforms": [
                {
                  "set": {
                    "value": "<our user>",
                    "target": "header.X-Auth-Email"
                  }
                },
                {
                  "set": {
                    "value": "<our auth key>",
                    "target": "header.X-Auth-Key"
                  }
                },
                {
                  "set": {
                    "default": """[[formatDate (now (parseDuration "-24h"))]]""",
                    "value": "[[.cursor.last_timestamp]]",
                    "target": "url.params.since"
                  }
                }
              ],
              "tags": [
                "forwarded",
                "cloudflare-audit",
                "preserve_original_event"
              ],
              "publisher_pipeline.disable_host": true
            },
            "data_stream": {
              "type": "logs",
              "dataset": "cloudflare.audit"
            },
            "vars": {
              "initial_interval": {
                "type": "text",
                "value": "24h"
              },
              "interval": {
                "type": "text",
                "value": "1h"
              },
              "auth_key": {
                "type": "password",
                "value": "<our auth key>"
              },
              "processors": {
                "type": "yaml"
              },
              "auth_email": {
                "type": "text",
                "value": "<our user>"
              },
              "account": {
                "type": "text",
                "value": "<our account id>"
              },
              "preserve_original_event": {
                "type": "bool",
                "value": true
              },
              "tags": {
                "type": "text",
                "value": [
                  "forwarded",
                  "cloudflare-audit"
                ]
              }
            },
            "id": "httpjson-cloudflare.audit-f5e823df-87d9-498f-bc49-3bc8fb08b5b4",
            "enabled": true
          },
          {
            "compiled_stream": {
              "cursor": {
                "last_execution_datetime": {
                  "value": """[[.last_response.url.params.Get "end"]]"""
                }
              },
              "response.decode_as": "application/x-ndjson",
              "request.method": "GET",
              "interval": "5m",
              "request.url": "https://api.cloudflare.com/client/v4/zones/<our zone id>/logs/received",
              "config_version": "2",
              "request.timeout": "60s",
              "request.transforms": [
                {
                  "set": {
                    "value": "<our user>",
                    "target": "header.X-Auth-Email"
                  }
                },
                {
                  "set": {
                    "value": "<our auth key>",
                    "target": "header.X-Auth-Key"
                  }
                },
                {
                  "set": {
                    "default": """[[formatDate (((now).Add (parseDuration "-1m")).Add (parseDuration "-5m"))]]""",
                    "value": "[[.cursor.last_execution_datetime]]",
                    "target": "url.params.start"
                  }
                },
                {
                  "set": {
                    "default": """[[formatDate ((now).Add (parseDuration "-1m"))]]""",
                    "value": """[[formatDate ((parseDate .cursor.last_execution_datetime).Add (parseDuration "5m"))]]""",
                    "target": "url.params.end"
                  }
                },
                {
                  "set": {
                    "value": "CacheCacheStatus,CacheResponseBytes,CacheResponseStatus,CacheTieredFill,ClientASN,ClientCountry,ClientDeviceType,ClientIP,ClientIPClass,ClientRequestBytes,ClientRequestHost,ClientRequestMethod,ClientRequestPath,ClientRequestProtocol,ClientRequestReferer,ClientRequestURI,ClientRequestUserAgent,ClientSSLCipher,ClientSSLProtocol,ClientSrcPort,ClientXRequestedWith,EdgeColoCode,EdgeColoID,EdgeEndTimestamp,EdgePathingOp,EdgePathingSrc,EdgePathingStatus,EdgeRateLimitAction,EdgeRateLimitID,EdgeRequestHost,EdgeResponseBytes,EdgeResponseCompressionRatio,EdgeResponseContentType,EdgeResponseStatus,EdgeServerIP,EdgeStartTimestamp,FirewallMatchesActions,FirewallMatchesRuleIDs,FirewallMatchesSources,OriginIP,OriginResponseBytes,OriginResponseHTTPExpires,OriginResponseHTTPLastModified,OriginResponseStatus,OriginResponseTime,OriginSSLProtocol,ParentRayID,RayID,SecurityLevel,WAFAction,WAFFlags,WAFMatchedVar,WAFProfile,WAFRuleID,WAFRuleMessage,WorkerCPUTime,WorkerStatus,WorkerSubrequest,WorkerSubrequestCount,ZoneID,Action",
                    "target": "url.params.fields"
                  }
                }
              ],
              "tags": [
                "forwarded",
                "cloudflare-logpull",
                "preserve_original_event"
              ],
              "publisher_pipeline.disable_host": true
            },
            "data_stream": {
              "type": "logs",
              "dataset": "cloudflare.logpull"
            },
            "vars": {
              "zone_id": {
                "type": "text",
                "value": "<our zone id>"
              },
              "interval": {
                "type": "text",
                "value": "5m"
              },
              "auth_key": {
                "type": "password",
                "value": "<our auth key>"
              },
              "processors": {
                "type": "yaml"
              },
              "auth_email": {
                "type": "text",
                "value": "<our user>"
              },
              "auth_token": {
                "type": "password",
                "value": ""
              },
              "preserve_original_event": {
                "type": "bool",
                "value": true
              },
              "tags": {
                "type": "text",
                "value": [
                  "forwarded",
                  "cloudflare-logpull"
                ]
              }
            },
            "id": "httpjson-cloudflare.logpull-f5e823df-87d9-498f-bc49-3bc8fb08b5b4",
            "enabled": true
          }
        ],
        "vars": {
          "api_url": {
            "type": "text",
            "value": "https://api.cloudflare.com"
          },
          "proxy_url": {
            "type": "text"
          },
          "ssl": {
            "type": "yaml"
          },
          "http_client_timeout": {
            "type": "text",
            "value": "60s"
          }
        },
        "type": "httpjson",
        "enabled": true
      }
    ],
    "revision": 31,
    "created_at": "2022-09-28T15:33:55.018Z",
    "created_by": "configuser",
    "updated_at": "2022-11-29T18:30:12.769Z",
    "updated_by": "configuser"
  }
}

```

---

<div class="post-metadata">

**Author:** ![ethhack](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethhack/32/109565_2.png) [@ethhack](https://discuss.elastic.co/u/ethhack)\
**Post date:** [November 29, 2022, 7:21pm UTC](https://discuss.elastic.co/t/another-cloudflare-integration-issue/320085/2 "2022-11-29T19:21:22Z")

</div>

Is it possible that .cursor.last\_execution\_datetime was the last time it successfully ran, and since that's now been more than a week, the pull is trying to parse a timeframe that is too old? (Similar to how Windows Active Directory API's do object change detection with a cookie value)

If so, is there an easy way to 'reset' it?

---

<div class="post-metadata">

**Author:** ![jamie.hynds](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jamie.hynds/32/84205_2.png) [@jamie.hynds](https://discuss.elastic.co/u/jamie.hynds)\
**Post date:** [November 29, 2022, 7:50pm UTC](https://discuss.elastic.co/t/another-cloudflare-integration-issue/320085/3 "2022-11-29T19:50:26Z")

</div>

Hi @ethhack - sorry to hear you’re running into an issue with the Cloudflare Logpull integration. @andrewkroh do you think this points to the cursor being set incorrectly on our end?

@ethhack out of curiosity, could you use Logpush as an alternative to Logpull? Based on our discussions with Cloudflare, Logpush is their preferred and recommended approach and we now have an integration with Logpush.

---

<div class="post-metadata">

**Author:** ![ethhack](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethhack/32/109565_2.png) [@ethhack](https://discuss.elastic.co/u/ethhack)\
**Post date:** [November 29, 2022, 7:57pm UTC](https://discuss.elastic.co/t/another-cloudflare-integration-issue/320085/4 "2022-11-29T19:57:16Z")

</div>

We're considering Logpush, longer-term, but based on some of our firewall requirements (regulatory), opening up listener ports to reach the Elastic Agent host means expanding some of our scope for audits, beyond what we'd 'like' to do, just to accommodate Cloudflare.

I've seen others doing manual pulls from the API and ingesting in their own way, but was hopeful that there's just be some easy way (for now, at least) to reset the cursor. (Maybe a PUT / POST from the API console of a value into the Elastic DB?)

I'm betting that if I were to whack and reinstall the Integration, it would wipe / reset the cursor and we'd start over. But this a has happened a couple of times before (where I did less homework to find the root cause and a reinstall DID work), and I don't want that to become the normal 'fix'.

---

<div class="post-metadata">

**Author:** ![ethhack](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethhack/32/109565_2.png) [@ethhack](https://discuss.elastic.co/u/ethhack)\
**Post date:** [December 5, 2022, 1:13pm UTC](https://discuss.elastic.co/t/another-cloudflare-integration-issue/320085/5 "2022-12-05T13:13:51Z")

</div>

Any updates / ideas here? We've now gone another week without being able to pull Cloudflare logs (it's been nearly a month since we were getting them reliably). I can pull them by hand, so there's that, but the goal is to have the data show up in some executive dashboards for some folks that have ZERO interest in accessing the data from anything other than the shared Elastic views.

---

<div class="post-metadata">

**Author:** ![ethhack](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethhack/32/109565_2.png) [@ethhack](https://discuss.elastic.co/u/ethhack)\
**Post date:** [December 12, 2022, 2:24pm UTC](https://discuss.elastic.co/t/another-cloudflare-integration-issue/320085/6 "2022-12-12T14:24:39Z")

</div>

Ping. Don’t want this going stale and closing because it’s not getting replied to. This is a substantial issue for us, and shouldn’t require uninstall / reinstall of the integration to resolve.

Can someone provide a solution or code edit to allow a reset here?

---

<div class="post-metadata">

**Author:** ![Marius\_Iversen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_iversen/32/68988_2.png) [@Marius\_Iversen](https://discuss.elastic.co/u/Marius_Iversen)\
**Post date:** [December 12, 2022, 3:51pm UTC](https://discuss.elastic.co/t/another-cloudflare-integration-issue/320085/7 "2022-12-12T15:51:28Z")

</div>

Hello @ethhack,  
Sorry for the long time before response, the forums are not really an official way to handle support requests, so we can only try to provide best effort help in terms of how long it takes for a response, especially now during holidays and such.

However let me see if I can answer your questions here!

1. For the Logpush part, we do support pushing logs to S3 and fetching them with agent (both with and without SQS), so if you feel that is a better long term solution then I would recommend that way.

2. For resetting the cursor, you would indeed have to remove the integration from the policy and re-add it again from Kibana to that host, or re-enroll the agent, or login to the machine itself and remove the cursor file from the state folder using by agent (and restart the agent running).  
The cursor is not stored on ES, so it wouldn't be possible to just do it with an API call, but you do have a few different options here.

When you got the issue with 168 hours, is that based on the agent not running for some time, and then being restarted? Once the initial\_interval has been used with the first API call, it will indeed use the cursor date always, so for example if the device has been down for a few days, and it hits the limitations of the Cloudflare API, then you would unfortunately see this issue.

---

<div class="post-metadata">

**Author:** ![ethhack](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethhack/32/109565_2.png) [@ethhack](https://discuss.elastic.co/u/ethhack)\
**Post date:** [December 12, 2022, 5:22pm UTC](https://discuss.elastic.co/t/another-cloudflare-integration-issue/320085/8 "2022-12-12T17:22:38Z")

</div>

No worries about the delayed response. I just didn’t want this one to fade away like the one I linked in the original request, before hopefully getting a look. We have enterprise support, as well, but have been otherwise preoccupied with other items, so when we saw the originally posted link, we thought we’d raise the issue here, again, in the hopes that someone had the answer by now.

Thanks for the info. I’ll give it all a look and report back. Was looking for the file to try to whack manually, and was unsuccessful so far.

The 168 hour was due to some communications issues we’d had, twice, with our clusters. Rare that we should hit that point, but still was an issue (especially if myself or other team member is on vacation or anything and this would happen, as it’s ONLY happening on the Cloudflare integration at present, so less noticeable to other team members).

Thanks again, and I’ll let you know how we fare.

---

<div class="post-metadata">

**Author:** ![ethhack](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethhack/32/109565_2.png) [@ethhack](https://discuss.elastic.co/u/ethhack)\
**Post date:** [December 12, 2022, 8:11pm UTC](https://discuss.elastic.co/t/another-cloudflare-integration-issue/320085/9 "2022-12-12T20:11:18Z")

</div>

Am I in the proper place if I edit the (path to my agent)/run/default/filebeat(blahblah..)/registry/filebeat/(myfile).json, find the most recent cursor for the failed collection, and edit the last\_execution\_datetime to a more recent time, then restart my agent / host?

I just did that, per your information above and I ‘think’ my events are starting to flow in again.

Assuming I’m correct, thank you for pointing me in the right direction for now (until we move to log push when we can), and hopefully this will help someone else with the same issues.

Much appreciated!

---

<div class="post-metadata">

**Author:** ![ethhack](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethhack/32/109565_2.png) [@ethhack](https://discuss.elastic.co/u/ethhack)\
**Post date:** [December 13, 2022, 3:20am UTC](https://discuss.elastic.co/t/another-cloudflare-integration-issue/320085/10 "2022-12-13T03:20:44Z")

</div>

Well… Assumed I still didn’t do enough by flipping the last\_execution\_datetime, only.

Looks like I also needed to set a valid start and end Unix timestamp (within the valid query timeframe) for updated, as well, perhaps?

Logs now show that I’m connecting with some occasional ‘repeated request’ errors, but that I am now receiving and publishing events.

Still not seeing some of what I’m used to in the dashboard, so I’ll monitor that tomorrow, as well, but looks like I’m making progress.

---

<div class="post-metadata">

**Author:** ![Marius\_Iversen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_iversen/32/68988_2.png) [@Marius\_Iversen](https://discuss.elastic.co/u/Marius_Iversen)\
**Post date:** [December 13, 2022, 9:07am UTC](https://discuss.elastic.co/t/another-cloudflare-integration-issue/320085/11 "2022-12-13T09:07:52Z")

</div>

Thanks for the update @ethhack, Glad you got the messages coming in again. Il monitor the topic for any updates if you have any more 👍

Usually you should not need to set any specific timestamps, as long as the line is completely removed it should start with the default `initial_interval` you have configured.

Any specific error messages you are getting for these repeated requests?

---

<div class="post-metadata">

**Author:** ![ethhack](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethhack/32/109565_2.png) [@ethhack](https://discuss.elastic.co/u/ethhack)\
**Post date:** [December 13, 2022, 12:12pm UTC](https://discuss.elastic.co/t/another-cloudflare-integration-issue/320085/12 "2022-12-13T12:12:25Z")

</div>

When I removed the line altogether and restarted, it still seemed stuck (nothing was moving, and I wasn’t seeing signs of fresh ingestion - unless we were having other issues with our agent that I was unaware of). Good to know that just removing the line should work (I may still test that today), as it’d be much easier than the extra, convoluted steps I posted, above.

I’ll let you know, but again, thanks for the pointers in the right direction. Far easier to do a quickie cleanup like this versus uninstalling / reinstalling or removing / re-adding the agent from the policy if it happens again.

---

<div class="post-metadata">

**Author:** ![ethhack](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethhack/32/109565_2.png) [@ethhack](https://discuss.elastic.co/u/ethhack)\
**Post date:** [December 13, 2022, 1:41pm UTC](https://discuss.elastic.co/t/another-cloudflare-integration-issue/320085/13 "2022-12-13T13:41:03Z")

</div>

Data anomaly is that something is now amiss with my ingest pipeline and mappings. Data is there, now, just not parsing correctly.

New day, new issue. But this one may be closed. The line delete worked great.

Thanks again for the assist.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 10, 2023, 1:41pm UTC](https://discuss.elastic.co/t/another-cloudflare-integration-issue/320085/14 "2023-01-10T13:41:23Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.