Invalid character 'x' in string escape code; pipeline/output.go:92 Failed to publish events: temporary bulk send failure

[adeel109 (Syed Adeel Ahmad)]

Hi All,

I am getting below error when starting filebeat:

2020-02-03T10:31:21.416Z ERROR reader/json.go:32 Error decoding JSON: invalid character 'x' in string escape code
2020-02-03T10:31:22.467Z INFO elasticsearch/client.go:690 Connected to Elasticsearch version 7.5.1

|2020-02-03T10:31:23.474Z|ERROR|pipeline/output.go:92|Failed to publish events: temporary bulk send failure|
|2020-02-03T10:31:23.475Z|INFO|elasticsearch/client.go:690|Connected to Elasticsearch version 7.5.1|
|2020-02-03T10:31:24.482Z|ERROR|pipeline/output.go:92|Failed to publish events: temporary bulk send failure|
|2020-02-03T10:31:24.484Z|INFO|elasticsearch/client.go:690|Connected to Elasticsearch version 7.5.1|
|2020-02-03T10:31:25.491Z|ERROR|pipeline/output.go:92|Failed to publish events: temporary bulk send failure|

filebeat config:

- type: log
  enabled: true
  paths:
  - /var/log/nginx/test4.json
  json:
    keys_under_root: true
    add_error_key: true
  fields:
    app_name: "nginx"
    hostname: "localhost"
  fields_under_root: true

output.elasticsearch:
  hosts: ["localhost:9200"]
  index: "test4-%{[fields.hostname]}-%{+YYYY}"
setup.template:
  name: "test4-%{[fields.hostname]}"
  pattern: "test4-%{[fields.hostname]}-*"
  enabled: false

log file format:

log_format json '{'
'"client_ip": "$remote_addr",'
'"client_port": "$remote_port",'
'"remote_user": "$remote_user",'
'"@timestamp": "$time_local",'
'"http_request": "$request",'
'"http_status_code": "$status",'
'"request_time": "$request_time",'
'"body_bytes_sent": "$body_bytes_sent",'
'"rh_http_referer": "$http_referer",'
'"agent": "$http_user_agent",'
'"http_x_forwarded_for": "$http_x_forwarded_for",'
'"upstream_response_time": "$upstream_response_time",'
'"uuid" : "$http_x_unique_id",'
'"authorization" : "$http_authorization",'
'"x_auth_timestamp" : "$http_x_auth_timestamp",'
'"data_version" : "$http_x_nfer_data",'
'"session_info" : "$http_cookie",'
'"user_email" : "$http_x_nfer_user",'
'"upstream_connect_time" : "$upstream_connect_time",'
'"upstream_response_time" : "$upstream_response_time",'
'"process_time" : "$request_time "'
'}';
log file:

{"client_ip": "1.2.3.4","client_port": "56268","remote_user": "-","@timestamp": "03/Feb/2020:09:45:47 +0000","http_request": "POST /endpoint/some/app HTTP/1.1","http_status_code": "200","request_time": "0.261","body_bytes_sent": "7362","rh_http_referer": "http://adeel.domain.com/results?query=some_query","agent": "python-requests/2.19.1","http_forwarded_for": "2.2.2.2","upstream_response_time": "0.261","uuid" : "2345236:w4t2346_343454236:346326","authorization" : "-","auth_timestamp" : "-","data_version" : "2","session_info" : "cookie","user_email" : "user@email","upstream_connect_time" : "0.001","upstream_response_time" : "0.261","process_time" : "0.261 "}

However, when I remove field data from output.elasticsearch index section, like below, I stiil get one error but the logs are pushed. I need to put field data in index name, please help.

- type: log
  enabled: true
  paths:
  - /var/log/nginx/test4.json
  json:
    keys_under_root: true
    add_error_key: true
  fields:
    app_name: "nginx"
    hostname: "localhost"
  fields_under_root: true

output.elasticsearch:
  hosts: ["localhost:9200"]
  index: "test4-%{+YYYY}"
setup.template:
  name: "test4"
  pattern: "test4-*"
  enabled: false

error

2020-02-03T10:52:02.351Z ERROR reader/json.go:32 Error decoding JSON: invalid character 'x' in string escape code

Despite this error, the logs are pushed.

My questions:

1. What does 'invalid character 'x' in string escape code' mean?
2. Why does including field in index name give temporary bulk send failure?
3. How do I fix this.

P.S.: I should mention that i am trying to push from filebeat version 6.2 to elasticsearch version 7.5