STARTDATE=$(($(date +%s)-3900))
ENDDATE=$((STARTDATE+3600))
curl -svo "$CLOUDFLARE_FILENAME" -X GET "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/logs/received?start=${STARTDATE}&end=${ENDDATE}×tamps=rfc3339&fields=$(curl -s -H "X-Auth-Email: $AUTH_EMAIL" -H "X-Auth-Key: $API_KEY" "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/logs/received/fields" | jq '. | to_entries[] | .key' -r | paste -sd "," -)" \
-H "X-Auth-Email: $AUTH_EMAIL" \
-H "X-Auth-Key: $API_KEY" \
-H "Content-Type: application/json";
```
_ **Logstash configuration: testcloudflare.conf** _
```auto
input {
file {
path => "/etc/logstash/testcloudflare/*.json"
sincedb_path => "/dev/null"
mode => "read"
#type => "gzip"
file_completed_action => "delete"
}
}
filter {
if [message] =~ "\A\{.+\}\z" {
json {
source => "message"
}
}
mutate {
add_field => {
"Full URL" => "https://%{ClientRequestHost}%{ClientRequestURI}"
}
}
geoip {
source => "OriginIP"
target => "server_geoip" }
geoip {
source => "ClientIP"
target => "geoip"
}
if "_jsonparsefailure" in [tags] {
drop { }
}
}
output {
elasticsearch {
hosts => ["http://192.168.10.29:9200"]
#ssl => true
#cacert => "/etc/logstash/certs/ca/ca.crt"
ilm_rollover_alias => "testcloudflare"
ilm_pattern => "{now{YYYY.MM.dd}}-000001"
ilm_policy => "testcloudflare-hot-warm-cold"
#user => "elastic"
#password => "elastic"
}
stdout {
codec => rubydebug
}
}
```
_ **Logstash pipelines.yml:** _
```auto
- pipeline.id: test
path.config: "/etc/logstash/conf.d/testcloudflare.conf"
```
During the testing, both of the script & configuration works fine but no output shown in Kibana. Is there something missing when ingesting Cloudflare logs inside logstash? Much appreciated. Thanks!
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [December 14, 2021, 6:16pm UTC](https://discuss.elastic.co/t/cloudflare-logs-via-logstash/291743/2 "2021-12-14T18:16:47Z")
If there are no events in Elasticsearch that suggest that logstash is dropping them all, which suggests that the json filter is not parsing them. I would remove that drop {} filter and take a look at the raw messages.
---
### Author:  [@ZiaulAfiq](https://discuss.elastic.co/u/ZiaulAfiq)
#### Post date: [December 16, 2021, 2:23am UTC](https://discuss.elastic.co/t/cloudflare-logs-via-logstash/291743/3 "2021-12-16T02:23:59Z")
Hey there @Badger! Thanks for the reply.
I have already commented out the drop {} filter but I still cannot found any raw messages. Is there any other way for me to see the output of the raw data?
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [December 16, 2021, 2:49am UTC](https://discuss.elastic.co/t/cloudflare-logs-via-logstash/291743/4 "2021-12-16T02:49:18Z")
Do you have anything in Logstash logs? How are you running logstash?
Are you running it as a service, since you are using `pipelines.yml` ? If so, does the `logstash` user has read/write permissions to the path `/etc/logstash/testcloudflare/`?
I have a similar pipeline, but I use a path where logstash has full permissions, which is not possible in `/etc`.
This is my input:
```auto
input {
file {
path => "/opt/logs-cloudflare/json/*.json"
sincedb_path => "/opt/logs-cloudflare/dbs/logs-cloudflare.db"
sincedb_clean_after => "2"
mode => "read"
file_completed_action => "delete"
}
}
```
Check Logstash logs to see if you have some errors and check if you have the right permissions, it doesn't seem to be anything wrong in the configuration.
---
### Author:  [@ZiaulAfiq](https://discuss.elastic.co/u/ZiaulAfiq)
#### Post date: [December 24, 2021, 8:24am UTC](https://discuss.elastic.co/t/cloudflare-logs-via-logstash/291743/5 "2021-12-24T08:24:16Z")
Hi there @leandrojmp. Yes I 'm running the Logstash as a service & the `logstash` user has read/write permissions to the path `/etc/logstash/testcloudflare/`.
As for the file path, `/etc/logstash/testcloudflare/*.json` return blank document when using `cat` command on the json file. Is the output for Cloudflare's Logpush should return a blank document on logstash?
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [December 24, 2021, 1:24pm UTC](https://discuss.elastic.co/t/cloudflare-logs-via-logstash/291743/6 "2021-12-24T13:24:43Z")
Do you have anything in Logstash logs that can give any hint of an issue?
If you having nothing in the source directory `/etc/logstash/testcloudflare/*.json` this can mean that Logstash is reading the file and deleting it or that your script is not working and is not creating the files.
You need to share Logstash logs to see if there is any hint of an issue.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 21, 2022, 1:24pm UTC](https://discuss.elastic.co/t/cloudflare-logs-via-logstash/291743/7 "2022-01-21T13:24:55Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.