Struggling here, grateful for any help

[lutonmad (Chris Johnson)]

Hi there,

So I'm probably confusing myself here but I am trying to import logs from our Cloudflare instance into Elasticsearch, via a file that has many json entries and looks like the following (this is just a single log)

{ "brandId":100,"flags":2,"hosterId":0,"ownerId":XXXXXX,"rayId":"XXXXXXXXX","securityLevel":"med","timestamp":1504731172284000000,"unstablePublic":null,"zoneId":XXXXXXX,"zoneName":"test.com","zonePlan":"enterprise","client":{"asNum":4648,"country":"XX","deviceType":"desktop","ip":"XXXXXXXX","ipClass":"noRecord","srcPort":63970,"sslCipher":"NONE","sslFlags":0,"sslProtocol":"none"},"clientRequest":{"accept":"text/html, application/xhtml+xml, /","body":null,"bodyBytes":0,"bytes":246,"cookies":null,"flags":1,"headers":,"httpHost":"www.test.com","httpMethod":"GET","httpProtocol":"HTTP/1.1","referer":"","uri":"/","userAgent":"Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"},"edge":{"bbResult":"0","cacheResponseTime":0,"colo":26,"enabledFlags":12,"endTimestamp":1504731172288000000,"flServerIp":"XXXXXXXX","flServerName":"26f27","flServerPort":80,"pathingOp":"wl","pathingSrc":"macro","pathingStatus":"nr","startTimestamp":1504731172284000000,"usedFlags":0,"rateLimit":{"ruleId":0,"mitigationId":null,"sourceId":"","processedRules":null},"dnsResponse":{"rcode":0,"error":"ok","cached":false,"duration":0,"errorMsg":"","overrideError":false}},"edgeResponse":{"bodyBytes":5,"bytes":285,"compressionRatio":0,"contentType":"","headers":null,"setCookies":null,"status":301}}

Originally I set up mappings as follows

curl -XPUT http://localhost:9200/cloudflare -d '
{
"mappings" : {
"default" : {
"properties" : {
"securityLevel" : {"type": "string" },
"zoneName" : {"type": "string" },
"cacheExternalIP" : {"type": "string" },
"cacheInternalIP" : {"type": "string" },
"cacheExternalPort" : { "type" : "integer" },
"ruleType" : { "type" : "integer" }
}
}
}
}
';

When I go to import the logs using curl -XPUT localhost:9200/cloudflare/_bulk --data-binary @cloudflare.json I get the following error message.

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Malformed action/metadata line [1], expected START_OBJECT or END_OBJECT but found [VALUE_NUMBER]"}],"type":"illegal_argument_exception","reason":"Malformed action/metadata line [1], expected START_OBJECT or END_OBJECT but found [VALUE_NUMBER]"},"status":400}

I did find this threat (_https://discuss.elastic.co/t/bulk-insert-file-having-many-json-entries-into-elasticsearch/46470_) and I understand the format is wrong when I am trying to insert using the Bulk format but I am not having much luck changing it.

Would anyone please be able to help me?

Thank you

[warkolm (Mark Walkom)]

This is what the format for a bulk post needs to look like - https://www.elastic.co/guide/en/elasticsearch/reference/5.6/docs-bulk.html

[mujtabahussain]

Hey!

So basically take one entry from the cloudflare.json file, and try and insert it into bulk format. That way, you can ensure that your data is correct and bulk format is working. Then try working a small subset of the data into bulk format and inserting! If that works as well, then do the full set. If that doesn't work, you know you have hit an issue with bulk inserting from the ES end and not your data or formatting

Try that and let us know

Best of luck

[lutonmad (Chris Johnson)]

Thanks, so this what I've done so far.

action_and_meta_data
{"brandId":100,"flags":2,"hosterId":0,"ownerId":4855861,"rayId":"XXXXXXXXXX","securityLevel":"med","timestamp":XXXXXXXXX,"unstablePublic":null,"zoneId":XXXXXX,"zoneName":"test.com","zonePlan":"enterprise","cache":{"bckType":"c3","cacheExternalIp":"","cacheExternalPort":0,"cacheFileKey":"XXXXXXXXX","cacheInternalIp":"XXXXXXX","cacheServerName":"12c161","cacheStatus":"hit","cacheTokens":0,"endTimestamp":1504669892158999808,"startTimestamp":1504669892158999808},"cacheResponse":{"bodyBytes":0,"bytes":2505,"contentType":"text/javascript","retriedStatus":0,"status":200},"client":{"asNum":15003,"country":"XX","deviceType":"desktop","ip":"104.238.45.55","ipClass":"noRecord","srcPort":50406,"sslCipher":"XXXXXXXXXXXX","sslFlags":1,"sslProtocol":"TLSv1.2"},"clientRequest":{"accept":"/","body":null,"bodyBytes":0,"bytes":1063,"cookies":null,"flags":1,"headers":,"httpHost":"www.test.com","httpMethod":"GET","httpProtocol":"HTTP/1.1","referer":"https://www.ehs.com/","uri":"/wp-content/themes/velocity/lib/js/resizer.js?ver=1.0.0","userAgent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36"},"edge":{"bbResult":"0","cacheResponseTime":5000000,"colo":12,"enabledFlags":12,"endTimestamp":1504669892160000000,"flServerIp":"104.20.7.52","flServerName":"12f190","flServerPort":443,"pathingOp":"wl","pathingSrc":"macro","pathingStatus":"nr","startTimestamp":1504669892155000064,"usedFlags":0,"rateLimit":{"ruleId":0,"mitigationId":null,"sourceId":"","processedRules":null},"dnsResponse":{"rcode":0,"error":"ok","cached":true,"duration":0,"errorMsg":"","overrideError":false}},"edgeRequest":{"bodyBytes":0,"bytes":1833,"headers":null,"httpHost":"www.test.com","httpMethod":"GET","keepaliveStatus":"reuseAccepted","uri":"/wp-content/themes/velocity/lib/js/resizer.js?ver=1.0.0"},"edgeResponse":{"bodyBytes":599,"bytes":1017,"compressionRatio":2.47,"contentType":"text/javascript","headers":null,"setCookies":null,"status":200}}

Which then gives me this error.

{"error":{"root_cause":[{"type":"parse_exception","reason":"Failed to derive xcontent"}],"type":"parse_exception","reason":"Failed to derive xcontent"},"status":400}

Any thoughts? Thanks in advance!

[warkolm (Mark Walkom)]

That's still not a valid bulk request.
You need to look lower in that doc page for the actual structure, what you have there is simple logical representation of the structure and not valid json.

[lutonmad (Chris Johnson)]

Thanks! I've reached out to Cloudflare for their assistance.