Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature]: Expose containers stats #1826

Open
pablochacin opened this issue Oct 27, 2023 · 1 comment · May be fixed by #1885
Open

[Feature]: Expose containers stats #1826

pablochacin opened this issue Oct 27, 2023 · 1 comment · May be fixed by #1885
Labels
feature New functionality or new behaviors on the existing one

Comments

@pablochacin
Copy link
Contributor

pablochacin commented Oct 27, 2023 •
edited

Problem

In some use cases, it would be convenient to have access to the container's stats to measure the resource (memory, CPU) consumption to detect regressions or establish a comparison between alternative implementations or configurations.

Solution

Docker provides this information in the stats command. The proposed solution is to expose this information from the Container interface as Container.Stats()

The raw information obtained from docker is quite extensive as seen in the example below, with many elements only available in Linux systems but not in Windows. Moreover, the units used for the reported metrics can vary between platforms. For example, Linux uses nanoseconds for measuring CPU time, while Windows uses 100s of nanoseconds.

container's stats in JSON format. Click to visualize 
  "stats": {
    "read": "2023-10-27T12:25:06.019724365Z",
    "pre_read": "0001-01-01T00:00:00Z",
    "pids_stats": {
      "current": 1059,
      "limit": 76608
    },
    "blkio_stats": {
      "io_service_bytes_recursive": [
        {
          "major": 253,
          "minor": 2,
          "op": "read",
          "value": 0
        },
        {
          "major": 253,
          "minor": 2,
          "op": "write",
          "value": 4096
        },
        {
          "major": 259,
          "minor": 0,
          "op": "read",
          "value": 1744896
        },
        {
          "major": 259,
          "minor": 0,
          "op": "write",
          "value": 0
        },
        {
          "major": 253,
          "minor": 0,
          "op": "read",
          "value": 1744896
        },
        {
          "major": 253,
          "minor": 0,
          "op": "write",
          "value": 12197888
        },
        {
          "major": 253,
          "minor": 1,
          "op": "read",
          "value": 1744896
        },
        {
          "major": 253,
          "minor": 1,
          "op": "write",
          "value": 12505088
        }
      ],
      "io_serviced_recursive": [],
      "io_queued_recursive": [],
      "io_service_time_recursive": [],
      "io_wait_time_recursive": [],
      "io_merged_recursive": [],
      "io_time_recursive": [],
      "sectors_recursive": []
    },
    "num_procs": 0,
    "storage_stats": {
      "read_count_normalized": 0,
      "read_size_bytes": 0,
      "write_count_normalized": 0,
      "write_size_bytes": 0
    },
    "cpu_stats": {
      "cpu_usage": {
        "total_usage": 25415300269000,
        "percpu_usage": [],
        "usage_in_kernelmode": 7710788995000,
        "usage_in_usermode": 17704511274000
      },
      "system_usage": 14395911570000000,
      "online_cpu_s": 20,
      "throttling_data": {
        "periods": 0,
        "throttled_periods": 0,
        "throttled_time": 0
      }
    },
    "pre_cpu_stats": {
      "cpu_usage": {
        "total_usage": 0,
        "percpu_usage": [],
        "usage_in_kernelmode": 0,
        "usage_in_usermode": 0
      },
      "system_usage": 0,
      "online_cpu_s": 0,
      "throttling_data": {
        "periods": 0,
        "throttled_periods": 0,
        "throttled_time": 0
      }
    },
    "memory_stats": {
      "usage": 3990077440,
      "max_usage": 0,
      "stats": {
        "pglazyfreed": 0,
        "pgmajfault": 341,
        "shmem": 275202048,
        "slab": 170641456,
        "anon_thp": 29360128,
        "file": 2074447872,
        "file_dirty": 4096,
        "sock": 12288,
        "file_mapped": 401707008,
        "file_writeback": 0,
        "pgactivate": 87325,
        "pgfault": 17291097,
        "pgrefill": 2955,
        "pgscan": 170011,
        "workingset_refault": 0,
        "active_file": 194707456,
        "anon": 1692143616,
        "workingset_nodereclaim": 0,
        "inactive_anon": 1813409792,
        "kernel_stack": 17350656,
        "unevictable": 0,
        "active_anon": 155107328,
        "inactive_file": 1606324224,
        "pgdeactivate": 2646,
        "slab_reclaimable": 144779624,
        "thp_collapse_alloc": 2,
        "thp_fault_alloc": 24,
        "pglazyfree": 436,
        "slab_unreclaimable": 25861832,
        "workingset_activate": 0,
        "pgsteal": 113971
      },
      "failcnt": 0,
      "limit": 67073630208,
      "commit": 0,
      "commit_peak": 0,
      "private_working_set": 0
    }
  },
  "read": "2023-10-27T12:25:06.019724365Z",
  "pre_read": "0001-01-01T00:00:00Z",
  "pids_stats": {
    "current": 1059,
    "limit": 76608
  },
  "blkio_stats": {
    "io_service_bytes_recursive": [
      {
        "major": 253,
        "minor": 2,
        "op": "read",
        "value": 0
      },
      {
        "major": 253,
        "minor": 2,
        "op": "write",
        "value": 4096
      },
      {
        "major": 259,
        "minor": 0,
        "op": "read",
        "value": 1744896
      },
      {
        "major": 259,
        "minor": 0,
        "op": "write",
        "value": 0
      },
      {
        "major": 253,
        "minor": 0,
        "op": "read",
        "value": 1744896
      },
      {
        "major": 253,
        "minor": 0,
        "op": "write",
        "value": 12197888
      },
      {
        "major": 253,
        "minor": 1,
        "op": "read",
        "value": 1744896
      },
      {
        "major": 253,
        "minor": 1,
        "op": "write",
        "value": 12505088
      }
    ],
    "io_serviced_recursive": [],
    "io_queued_recursive": [],
    "io_service_time_recursive": [],
    "io_wait_time_recursive": [],
    "io_merged_recursive": [],
    "io_time_recursive": [],
    "sectors_recursive": []
  },
  "num_procs": 0,
  "storage_stats": {
    "read_count_normalized": 0,
    "read_size_bytes": 0,
    "write_count_normalized": 0,
    "write_size_bytes": 0
  },
  "cpu_stats": {
    "cpu_usage": {
      "total_usage": 25415300269000,
      "percpu_usage": [],
      "usage_in_kernelmode": 7710788995000,
      "usage_in_usermode": 17704511274000
    },
    "system_usage": 14395911570000000,
    "online_cpu_s": 20,
    "throttling_data": {
      "periods": 0,
      "throttled_periods": 0,
      "throttled_time": 0
    }
  },
  "pre_cpu_stats": {
    "cpu_usage": {
      "total_usage": 0,
      "percpu_usage": [],
      "usage_in_kernelmode": 0,
      "usage_in_usermode": 0
    },
    "system_usage": 0,
    "online_cpu_s": 0,
    "throttling_data": {
      "periods": 0,
      "throttled_periods": 0,
      "throttled_time": 0
    }
  },
  "memory_stats": {
    "usage": 3990077440,
    "max_usage": 0,
    "stats": {
      "inactive_anon": 1813409792,
      "kernel_stack": 17350656,
      "unevictable": 0,
      "active_anon": 155107328,
      "inactive_file": 1606324224,
      "pgdeactivate": 2646,
      "slab_reclaimable": 144779624,
      "thp_collapse_alloc": 2,
      "thp_fault_alloc": 24,
      "pglazyfree": 436,
      "slab_unreclaimable": 25861832,
      "workingset_activate": 0,
      "pgsteal": 113971,
      "pglazyfreed": 0,
      "pgmajfault": 341,
      "shmem": 275202048,
      "slab": 170641456,
      "anon_thp": 29360128,
      "file": 2074447872,
      "file_dirty": 4096,
      "sock": 12288,
      "file_mapped": 401707008,
      "file_writeback": 0,
      "pgactivate": 87325,
      "pgfault": 17291097,
      "pgrefill": 2955,
      "pgscan": 170011,
      "workingset_refault": 0,
      "active_file": 194707456,
      "anon": 1692143616,
      "workingset_nodereclaim": 0
    },
    "failcnt": 0,
    "limit": 67073630208,
    "commit": 0,
    "commit_peak": 0,
    "private_working_set": 0
  },
  "name": "/test-e2e-control-plane",
  "id": "6271ed069235600bcbdbe70505db90f299197c69a49b097dfa5b6e941b917172",
  "networks": {
    "eth0": {
      "rx_bytes": 80197969,
      "rx_packets": 19779,
      "rx_errors": 0,
      "rx_dropped": 0,
      "tx_bytes": 2127699,
      "tx_packets": 5507,
      "tx_errors": 0,
      "tx_dropped": 0,
      "endpoint_id": "",
      "instance_id": ""
    }
  }
}

Therefore, we suggest returning a custom type with the information provided by the stats command and containing only fields that are available in both Linux and Windows platforms and normalized to one unit (for example, nanoseconds):

type ContainerStats struct {
            Timestamp               time.Time
            CPUUsageTotal       uint64
            CPUUsageInKernel uint64
            CPUUsageUser       uint64
            CPUPercentage      float64
            MemoryUsage         uint64               // in Windows, it corresponds to Commit memory
            MemoryMaxUsage   uint64               // in Windows, it corresponds to PeakCommit memory
            MemoryPercentage float64
            NetworkRxBytes      uint64
            NetworkTxBytes      uint64
            BlockIOReadBytes  uint64               // BlockIO metrics must be calculated from different sources in Windows and Linux
            BlockIOWriteBytes  uint64                
}

Benefit

Offers the ability to monitor a container's resource usage.

Alternatives

We don't see any alternative that is convenient for the test developer.

Would you like to help contributing this feature?

Yes
@pablochacin pablochacin added the feature New functionality or new behaviors on the existing one label Oct 27, 2023
@pablochacin pablochacin linked a pull request Nov 2, 2023 that will close this issue
Draft
@ervitis
Copy link

ervitis commented May 1, 2024

hi @pablochacin! I started using testcontainers-go and I would be very interested in using this feature

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature New functionality or new behaviors on the existing one
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

WIP: PoC Container Stats pablochacin/testcontainers-go
2 participants
@pablochacin @ervitis