awslogs batch size exceeds service limits

[jahkeup]

Description

awslogs driver improperly batches log messages that are too large for CloudWatch to accept. This is due to incorrect accounting of the total size of the message being logged. The overhead and total byte size calculation is not maintained between the scopes that evaluate the service limit.

I have prepared a patch to fix this issue and will open a pull request link here.

Steps to reproduce the issue:

1. Run docker container with awslogs driver.
2. Log a message larger than the allowed limit (1048576), such as 1048685 bytes from that container.
3. Observe docker log output.

Describe the results you received:

Container log messages are not logged to CloudWatch and are instead dropped by the driver due to an "exception" (error). This is error is printed in the Docker log:

time="2017-12-06T23:34:21.421389421Z" level=error msg="Failed to put log events" errorCode=InvalidParameterException logGroupName=cbcab4da0300039d1d8e10b40b053ea5 logStreamName="klines/klines/5c8ae105-9de2-40ab-ab8d-7b0bfd46a01a" message="Upload too large: 1048815 bytes exceeds limit of 1048576" origError=<nil> 
time="2017-12-06T23:34:21.421436838Z" level=error msg="InvalidParameterException: Upload too large: 1048815 bytes exceeds limit of 1048576\n\tstatus code: 400, request id: fc4705d1-dadd-11e7-8ef0-21e93a30f134" 

Describe the results you expected:

I expected the log messages to be split as needed to log the message to CloudWatch Logs.

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

Client:
 Version:      17.06.2-ce
 API version:  1.30
 Go version:   go1.8.4
 Git commit:   3dfb8343b139d6342acfd9975d7f1068b5b1c3d3
 Built:        Fri Nov 10 00:50:37 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.06.2-ce
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.4
 Git commit:   402dd4a/17.06.2-ce
 Built:        Fri Nov 10 00:51:08 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Containers: 3
 Running: 1
 Paused: 0
 Stopped: 2
Images: 3
Server Version: 17.06.2-ce
Storage Driver: devicemapper
 Pool Name: docker-docker--pool
 Pool Blocksize: 524.3kB
 Base Device Size: 10.74GB
 Backing Filesystem: ext4
 Data file: 
 Metadata file: 
 Data Space Used: 1.239GB
 Data Space Total: 23.33GB
 Data Space Available: 22.09GB
 Metadata Space Used: 593.9kB
 Metadata Space Total: 25.17MB
 Metadata Space Available: 24.57MB
 Thin Pool Minimum Free Space: 2.333GB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.135-RHEL7 (2016-11-16)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 6e23458c129b551d5c9871e5174f6b1b7f6d1170
runc version: 810190ceaa507aa2727d7ae6f4790c76ec150bd2
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.62-21.56.amzn1.x86_64
Operating System: Amazon Linux AMI 2017.09
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.955GiB
Name: ip-172-31-38-65
ID: XNRA:RMA2:5VQ5:NLVZ:WAWS:VJFF:24R7:RZNS:32X4:DRF2:SA2S:WP7C
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):
AWS on a t2.small

[alexthornton]

This is also affects us. Thank you and others reviewing for the patch!