Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

[馃悰 Bug]: Videos no longer recorded #2156

Closed
MJB222398 opened this issue Feb 29, 2024 · 9 comments
Closed

[馃悰 Bug]: Videos no longer recorded #2156

MJB222398 opened this issue Feb 29, 2024 · 9 comments

Comments

@MJB222398
Copy link

What happened?

I use a selenium grid with 3 browser nodes (chrome, edge, and firefox) and have a video container for each node to record them. I spin these all up using a docker compose. All was working fine with version 4.16.1-20231219 and ffmpeg-6.1-20231219. But now I upgraded to 4.18.0-20240220 and ffmpeg-6.1-20240220 and now videos are no longer recorded.

Command used to start Selenium Grid with Docker (or Kubernetes)

# To execute this docker-compose yml file use `docker-compose -f docker-compose.yml up`
# Add the `-d` flag at the end for detached execution
# To stop the execution, hit Ctrl+C, and then `docker-compose -f docker-compose.yml down`
version: "3"
services:
  chrome:
    image: selenium/node-chrome:4.18.0-20240220
    container_name: chrome-node
    shm_size: 2gb
    privileged: true
    depends_on:
      - selenium-hub
    environment:
      - HUB_HOST=selenium-hub
      - SE_EVENT_BUS_HOST=selenium-hub
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443
      - SE_NODE_MAX_SESSIONS=1
      - SE_NODE_GRID_URL=http://localhost:4444
      - SE_NODE_SESSION_TIMEOUT=180
    ports:
      - "6900:5900"
    volumes:
      - ./chrome:/selenium/chrome
    command:
      - /bin/bash
      - -c
      - |
        sh /selenium/chrome/profile-setup.sh
        sh /opt/bin/entry_point.sh
    networks:
      - selenium_grid_internal

  edge:
    image: selenium/node-edge:4.18.0-20240220
    container_name: edge-node
    shm_size: 2gb
    privileged: true
    depends_on:
      - selenium-hub
    environment:
      - HUB_HOST=selenium-hub
      - SE_EVENT_BUS_HOST=selenium-hub
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443
      - SE_NODE_MAX_SESSIONS=1
      - SE_NODE_GRID_URL=http://localhost:4444
      - SE_NODE_SESSION_TIMEOUT=180
    ports:
      - "6901:5900"
    volumes:
      - ./edge:/selenium/edge
    command:
      - /bin/bash
      - -c
      - |
        sh /selenium/edge/profile-setup.sh
        sh /opt/bin/entry_point.sh
    networks:
      - selenium_grid_internal

  firefox:
    image: selenium/node-firefox:4.18.0-20240220
    container_name: firefox-node
    shm_size: 2gb
    privileged: true
    depends_on:
      - selenium-hub
    environment:
      - HUB_HOST=selenium-hub
      - SE_EVENT_BUS_HOST=selenium-hub
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443
      - SE_NODE_MAX_SESSIONS=1
      - SE_NODE_GRID_URL=http://localhost:4444
      - SE_NODE_SESSION_TIMEOUT=180
    ports:
      - "6902:5900"
    volumes:
      - ./firefox:/selenium/firefox
    command:
      - /bin/bash
      - -c
      - |
        sh /selenium/firefox/profile-setup.sh
        sh /opt/bin/entry_point.sh
    networks:
      - selenium_grid_internal

  chrome_video:
    image: selenium/video:ffmpeg-6.1-20240220
    container_name: chrome-video
    privileged: true
    volumes:
      - ${VIDEOS_DIRECTORY:-./videos}:/videos
    depends_on:
      - chrome
    environment:
      - DISPLAY_CONTAINER_NAME=chrome
      - FILE_NAME=chrome_video.mp4
    networks:
      - selenium_grid_internal

  edge_video:
    image: selenium/video:ffmpeg-6.1-20240220
    container_name: edge-video
    privileged: true
    volumes:
      - ${VIDEOS_DIRECTORY:-./videos}:/videos
    depends_on:
      - edge
    environment:
      - DISPLAY_CONTAINER_NAME=edge
      - FILE_NAME=edge_video.mp4
    networks:
      - selenium_grid_internal

  firefox_video:
    image: selenium/video:ffmpeg-6.1-20240220
    container_name: firefox-video
    privileged: true
    volumes:
      - ${VIDEOS_DIRECTORY:-./videos}:/videos
    depends_on:
      - firefox
    environment:
      - DISPLAY_CONTAINER_NAME=firefox
      - FILE_NAME=firefox_video.mp4
    networks:
      - selenium_grid_internal

  selenium-hub:
    image: selenium/hub:4.18.0-20240220
    privileged: true
    container_name: selenium-hub
    hostname: selenium-hub
    ports:
      - "4442:4442"
      - "4443:4443"
      - "4444:4444"
    environment:
      - GRID_TIMEOUT=120000
      - GRID_BROWSER_TIMEOUT=120000
    networks:
      - selenium_grid_internal

networks:
  selenium_grid_internal:

Relevant log output

*  Executing task: docker logs --tail 1000 -f 98b1c1bb91e67afdef394056288672aab967db2ac3469f39f7b49503c647b0e0 

2024-02-29 12:31:48,096 INFO supervisord started with pid 8
2024-02-29 12:31:49,098 INFO spawned: 'video-recording' with pid 9
2024-02-29 12:31:49,099 INFO spawned: 'video-ready' with pid 10
Checking if the display is open...
Waiting before next display check...
Waiting before next display check...
Waiting before next display check...
2024-02-29 12:31:50,106 INFO success: video-recording entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-02-29 12:31:50,106 INFO success: video-ready entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Waiting before next display check...
Waiting before next display check...
Waiting before next display check...
Waiting before next display check...
Waiting before next display check...
Waiting before next display check...
Waiting before next display check...
Waiting before next display check...
Waiting before next display check...
Waiting before next display check...
Waiting before next display check...
Waiting before next display check...
Waiting before next display check...
Waiting before next display check...
Waiting before next display check...
Waiting before next display check...
Waiting before next display check...
Waiting before next display check...
Waiting before next display check...
Waiting before next display check...
Waiting before next display check...
Waiting before next display check...
Waiting before next display check...
ffmpeg version 6.1.1 Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 11 (Ubuntu 11.4.0-1ubuntu1~22.04)
  configuration: --disable-debug --disable-doc --disable-ffplay --enable-fontconfig --enable-gpl --enable-libaom --enable-libaribb24 --enable-libass --enable-libbluray --enable-libfdk_aac --enable-libfreetype --enable-libkvazaar --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libsrt --enable-libtheora --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxvid --enable-libzmq --enable-nonfree --enable-openssl --enable-postproc --enable-shared --enable-small --enable-version3 --extra-cflags=-I/opt/ffmpeg/include --extra-ldflags=-L/opt/ffmpeg/lib --extra-libs=-ldl --extra-libs=-lpthread --prefix=/opt/ffmpeg
  libavutil      58. 29.100 / 58. 29.100
  libavcodec     60. 31.102 / 60. 31.102
  libavformat    60. 16.100 / 60. 16.100
  libavdevice    60.  3.100 / 60.  3.100
  libavfilter     9. 12.100 /  9. 12.100
  libswscale      7.  5.100 /  7.  5.100
  libswresample   4. 12.100 /  4. 12.100
  libpostproc    57.  3.100 / 57.  3.100
[x11grab @ 0x55a2fd211a40] Cannot get the image data event_error: response_type:0 error_code:1 sequence:9 resource_id:1293 minor_code:4 major_code:130.
[x11grab @ 0x55a2fd211a40] Continuing without shared memory.
[x11grab @ 0x55a2fd211a40] Stream #0: not enough frames to estimate rate; consider increasing probesize
Input #0, x11grab, from 'chrome:99.0':
  Duration: N/A, start: 1709209922.225129, bitrate: 665856 kb/s
  Stream #0:0: Video: rawvideo (BGR[0] / 0x524742), bgr0, 1360x1020, 665856 kb/s, 15 fps, 1000k tbr, 1000k tbn
[out#0/mp4 @ 0x55a2fd21cc00] Error opening output /videos/chrome_video.mp4: Permission denied
Error opening output file /videos/chrome_video.mp4.
Error opening output files: Permission denied
2024-02-29 12:32:02,228 INFO exited: video-recording (exit status 243; not expected)
2024-02-29 12:32:03,230 INFO spawned: 'video-recording' with pid 65
Checking if the display is open...
ffmpeg version 6.1.1 Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 11 (Ubuntu 11.4.0-1ubuntu1~22.04)
  configuration: --disable-debug --disable-doc --disable-ffplay --enable-fontconfig --enable-gpl --enable-libaom --enable-libaribb24 --enable-libass --enable-libbluray --enable-libfdk_aac --enable-libfreetype --enable-libkvazaar --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libsrt --enable-libtheora --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxvid --enable-libzmq --enable-nonfree --enable-openssl --enable-postproc --enable-shared --enable-small --enable-version3 --extra-cflags=-I/opt/ffmpeg/include --extra-ldflags=-L/opt/ffmpeg/lib --extra-libs=-ldl --extra-libs=-lpthread --prefix=/opt/ffmpeg
  libavutil      58. 29.100 / 58. 29.100
  libavcodec     60. 31.102 / 60. 31.102
  libavformat    60. 16.100 / 60. 16.100
  libavdevice    60.  3.100 / 60.  3.100
  libavfilter     9. 12.100 /  9. 12.100
  libswscale      7.  5.100 /  7.  5.100
  libswresample   4. 12.100 /  4. 12.100
  libpostproc    57.  3.100 / 57.  3.100
[x11grab @ 0x5566ec29ba40] Cannot get the image data event_error: response_type:0 error_code:1 sequence:9 resource_id:1293 minor_code:4 major_code:130.
[x11grab @ 0x5566ec29ba40] Continuing without shared memory.
[x11grab @ 0x5566ec29ba40] Stream #0: not enough frames to estimate rate; consider increasing probesize
Input #0, x11grab, from 'chrome:99.0':
  Duration: N/A, start: 1709209923.306779, bitrate: 665856 kb/s
  Stream #0:0: Video: rawvideo (BGR[0] / 0x524742), bgr0, 1360x1020, 665856 kb/s, 15 fps, 1000k tbr, 1000k tbn
[out#0/mp4 @ 0x5566ec2a6c00] Error opening output /videos/chrome_video.mp4: Permission denied
Error opening output file /videos/chrome_video.mp4.
Error opening output files: Permission denied
2024-02-29 12:32:03,310 INFO exited: video-recording (exit status 243; not expected)
2024-02-29 12:32:04,312 INFO spawned: 'video-recording' with pid 68
Checking if the display is open...
ffmpeg version 6.1.1 Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 11 (Ubuntu 11.4.0-1ubuntu1~22.04)
  configuration: --disable-debug --disable-doc --disable-ffplay --enable-fontconfig --enable-gpl --enable-libaom --enable-libaribb24 --enable-libass --enable-libbluray --enable-libfdk_aac --enable-libfreetype --enable-libkvazaar --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libsrt --enable-libtheora --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxvid --enable-libzmq --enable-nonfree --enable-openssl --enable-postproc --enable-shared --enable-small --enable-version3 --extra-cflags=-I/opt/ffmpeg/include --extra-ldflags=-L/opt/ffmpeg/lib --extra-libs=-ldl --extra-libs=-lpthread --prefix=/opt/ffmpeg
  libavutil      58. 29.100 / 58. 29.100
  libavcodec     60. 31.102 / 60. 31.102
  libavformat    60. 16.100 / 60. 16.100
  libavdevice    60.  3.100 / 60.  3.100
  libavfilter     9. 12.100 /  9. 12.100
  libswscale      7.  5.100 /  7.  5.100
  libswresample   4. 12.100 /  4. 12.100
  libpostproc    57.  3.100 / 57.  3.100
[x11grab @ 0x555ec7472a40] Cannot get the image data event_error: response_type:0 error_code:1 sequence:9 resource_id:1293 minor_code:4 major_code:130.
[x11grab @ 0x555ec7472a40] Continuing without shared memory.
[x11grab @ 0x555ec7472a40] Stream #0: not enough frames to estimate rate; consider increasing probesize
Input #0, x11grab, from 'chrome:99.0':
  Duration: N/A, start: 1709209924.387853, bitrate: 665856 kb/s
  Stream #0:0: Video: rawvideo (BGR[0] / 0x524742), bgr0, 1360x1020, 665856 kb/s, 15 fps, 1000k tbr, 1000k tbn
[out#0/mp4 @ 0x555ec747dc00] Error opening output /videos/chrome_video.mp4: Permission denied
Error opening output file /videos/chrome_video.mp4.
Error opening output files: Permission denied
2024-02-29 12:32:04,391 INFO exited: video-recording (exit status 243; not expected)
2024-02-29 12:32:06,394 INFO spawned: 'video-recording' with pid 71
Checking if the display is open...
ffmpeg version 6.1.1 Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 11 (Ubuntu 11.4.0-1ubuntu1~22.04)
  configuration: --disable-debug --disable-doc --disable-ffplay --enable-fontconfig --enable-gpl --enable-libaom --enable-libaribb24 --enable-libass --enable-libbluray --enable-libfdk_aac --enable-libfreetype --enable-libkvazaar --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libsrt --enable-libtheora --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxvid --enable-libzmq --enable-nonfree --enable-openssl --enable-postproc --enable-shared --enable-small --enable-version3 --extra-cflags=-I/opt/ffmpeg/include --extra-ldflags=-L/opt/ffmpeg/lib --extra-libs=-ldl --extra-libs=-lpthread --prefix=/opt/ffmpeg
  libavutil      58. 29.100 / 58. 29.100
  libavcodec     60. 31.102 / 60. 31.102
  libavformat    60. 16.100 / 60. 16.100
  libavdevice    60.  3.100 / 60.  3.100
  libavfilter     9. 12.100 /  9. 12.100
  libswscale      7.  5.100 /  7.  5.100
  libswresample   4. 12.100 /  4. 12.100
  libpostproc    57.  3.100 / 57.  3.100
[x11grab @ 0x55deb5cb7a40] Cannot get the image data event_error: response_type:0 error_code:1 sequence:9 resource_id:1293 minor_code:4 major_code:130.
[x11grab @ 0x55deb5cb7a40] Continuing without shared memory.
[x11grab @ 0x55deb5cb7a40] Stream #0: not enough frames to estimate rate; consider increasing probesize
Input #0, x11grab, from 'chrome:99.0':
  Duration: N/A, start: 1709209926.469987, bitrate: 665856 kb/s
  Stream #0:0: Video: rawvideo (BGR[0] / 0x524742), bgr0, 1360x1020, 665856 kb/s, 15 fps, 1000k tbr, 1000k tbn
[out#0/mp4 @ 0x55deb5cc2c00] Error opening output /videos/chrome_video.mp4: Permission denied
Error opening output file /videos/chrome_video.mp4.
Error opening output files: Permission denied
2024-02-29 12:32:06,473 INFO exited: video-recording (exit status 243; not expected)
2024-02-29 12:32:09,477 INFO spawned: 'video-recording' with pid 74
Checking if the display is open...
ffmpeg version 6.1.1 Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 11 (Ubuntu 11.4.0-1ubuntu1~22.04)
  configuration: --disable-debug --disable-doc --disable-ffplay --enable-fontconfig --enable-gpl --enable-libaom --enable-libaribb24 --enable-libass --enable-libbluray --enable-libfdk_aac --enable-libfreetype --enable-libkvazaar --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libsrt --enable-libtheora --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxvid --enable-libzmq --enable-nonfree --enable-openssl --enable-postproc --enable-shared --enable-small --enable-version3 --extra-cflags=-I/opt/ffmpeg/include --extra-ldflags=-L/opt/ffmpeg/lib --extra-libs=-ldl --extra-libs=-lpthread --prefix=/opt/ffmpeg
  libavutil      58. 29.100 / 58. 29.100
  libavcodec     60. 31.102 / 60. 31.102
  libavformat    60. 16.100 / 60. 16.100
  libavdevice    60.  3.100 / 60.  3.100
  libavfilter     9. 12.100 /  9. 12.100
  libswscale      7.  5.100 /  7.  5.100
  libswresample   4. 12.100 /  4. 12.100
  libpostproc    57.  3.100 / 57.  3.100
[x11grab @ 0x5557ffbbda40] Cannot get the image data event_error: response_type:0 error_code:1 sequence:9 resource_id:1293 minor_code:4 major_code:130.
[x11grab @ 0x5557ffbbda40] Continuing without shared memory.
[x11grab @ 0x5557ffbbda40] Stream #0: not enough frames to estimate rate; consider increasing probesize
Input #0, x11grab, from 'chrome:99.0':
  Duration: N/A, start: 1709209929.553416, bitrate: 665856 kb/s
  Stream #0:0: Video: rawvideo (BGR[0] / 0x524742), bgr0, 1360x1020, 665856 kb/s, 15 fps, 1000k tbr, 1000k tbn
[out#0/mp4 @ 0x5557ffbc8c00] Error opening output /videos/chrome_video.mp4: Permission denied
Error opening output file /videos/chrome_video.mp4.
Error opening output files: Permission denied
2024-02-29 12:32:09,556 INFO exited: video-recording (exit status 243; not expected)
2024-02-29 12:32:10,558 INFO gave up: video-recording entered FATAL state, too many start retries too quickly

Operating System

Ubuntu 20.04

Docker Selenium version (image tag)

4.18.0-20240220

Selenium Grid chart version (chart version)

No response

Copy link

@MJB222398, thank you for creating this issue. We will troubleshoot it as soon as we can.


Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

@VietND96
Copy link
Member

@MJB222398, before launching containers, can you update steps as below

mkdir -p ${VIDEOS_DIRECTORY:-./videos}
sudo chown 1200:1201 ${VIDEOS_DIRECTORY:-./videos}
docker compose up -d

This is needed after #2122
It is mentioned in https://github.com/SeleniumHQ/docker-selenium?tab=readme-ov-file#mounting-volumes-to-retrieve-video-files

@MJB222398
Copy link
Author

MJB222398 commented Feb 29, 2024

@VietND96 thanks that does seem to have worked locally for me. Getting that to work on the CICD machine might be more of an issue... But why does #2122 say that it's not a breaking change? If that hasn't broken my process then what has? Also why is having to run the above command before spinning up containers a good thing? Maybe I am missing something but it worked before with less effort.

@VietND96
Copy link
Member

It looks like an issue is with linux, if the folder is not present, it will be created by root, and before the container runs as root by default, so the issue could not be seen.
We also have CI tests, here, at least we have to pre-create the folder and run the container with UID same as host UID by docker run -u or user: specified in docker-compose

rm -rf ./tests/videos; mkdir -p ./tests/videos


Can you check is this help in your CICD?

@VietND96 VietND96 pinned this issue Feb 29, 2024
@VietND96
Copy link
Member

In case you don't worry about rootless for this container, I think you can specify user: 0 in docker-compose to let it run as root. It would be less effort same as before.

@MJB222398
Copy link
Author

@VietND96 I cannot write user: 0 in the video container in the docker-compose. If I do I get this error:

ERROR: The Compose file './docker-compose.yml' is invalid because:
services.chrome_video.user contains an invalid type, it should be a string
services.edge_video.user contains an invalid type, it should be a string
services.firefox_video.user contains an invalid type, it should be a string

So I wrote user: root instead and this seems to work.

If I instead use user: ${UID} then I get this error when spinning up containers:

WARNING: The UID variable is not set. Defaulting to a blank string.

but if I do an echo $UID from the same shell it will print out my UID correctly

@VietND96
Copy link
Member

VietND96 commented Mar 1, 2024

${UID} in docker-compose file could not fetch value itself. It need a dotenv .env file in the same directory. You need to perform commands like

echo UID=$UID > .env
docker compose up -d

So, which way is suitable for you?

@MJB222398
Copy link
Author

@VietND96 Thanks for your help, I didn't realise that not all environment variables are made available automatically (my VIDEOS_DIRECTORY one is for instance). All working now using UID locally and on my CICD machine.

@VietND96 VietND96 closed this as completed Mar 4, 2024
Copy link

github-actions bot commented Apr 4, 2024

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked and limited conversation to collaborators Apr 4, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

2 participants