[Bug]: frame extraction by timestamp operation can be very slow #44

humpydonkey · 2024-03-21T20:39:53Z

Description

This recipe demonstrated extracting a single frame by a timestamp via the FFdecoder API.
However, the frame extraction time goes up when the timestamp gets bigger.
The extraction time is so slow such that this API is not usable for my use case. Thus, filing a bug report for an expedited fix.

Looked deeper into the generated ffmpeg command, I found that the generated command is not optimal. A better approach is to use Input Seeking instead of Output Seeking, ref.

Here is an example generated command:

ffmpeg -vcodec hevc -i /Users/asia/Downloads/frames/HacksawRidge.2016.BluRay.1080p.x265.10bit.2Audio-MiniHD.mkv -pix_fmt rgb24 -s 1920x804 -framerate 23.98 -ss 00:30:00.00 -t 120s -r 0.5 -f rawvideo -

Notice that the -ss flag is placed after the input file, i.e. -i video_file_path.

If we place the -ss flag before the input file, this operation becomes a lot more faster. E.g.

ffmpeg -vcodec hevc ss 00:30:00.00 -t 120s -i /Users/asia/Downloads/frames/HacksawRidge.2016.BluRay.1080p.x265.10bit.2Audio-MiniHD.mkv -pix_fmt rgb24 -s 1920x804 -framerate 23.98 -r 0.5 -f rawvideo -

Changing the ordering of the flag should be a quick fix for this issue.
Hoping we could get it fixed soon.

Issue Checklist

I have searched open or closed issues for my problem and found nothing related or helpful.
I have read the Documentation and found nothing related to my problem.
I've read the Issue Guidelines and wholeheartedly agree.

Expected behaviour

Extracting a single frame by a timestamp via the FFdecoder API should be fast regardless of the timestamp.

Actual behaviour

The extraction time is so slow such that this API is not usable for my use case.

Steps to reproduce

from deffcode import FFdecoder
from PIL import Image
from pathlib import Path

video_path = "~/Downloads/frames/HacksawRidge.2016.BluRay.1080p.x265.10bit.2Audio-MiniHD.mkv"

ffparams = {
    "-ss": "00:30:00.00",
    "-t": "120s",
    "-r": 2,
}

output = "output"
decoder = FFdecoder(video_path, verbose=True, **ffparams).formulate()
frame_count = 0

for frame in decoder.generateFrame():
    if frame is None:
        break
    frame_count += 1
    Image.fromarray(frame).save(f"{output}/frame_{frame_count}.jpeg")

# terminate the decoder
decoder.terminate()

Terminal log output

�[32m13:12:27�[0m :: �[1;35m  FFhelper   �[0m :: �[1;33m DEBUG  �[0m :: �[1;37mFinal FFmpeg Path: ffmpeg�[0m
�[32m13:12:28�[0m :: �[1;35m  FFhelper   �[0m :: �[1;33m DEBUG  �[0m :: �[1;37mFFmpeg validity Test Passed!�[0m
�[32m13:12:28�[0m :: �[1;35m  FFhelper   �[0m :: �[1;33m DEBUG  �[0m :: �[1;37mFound valid FFmpeg Version: `b'6.0'` installed on this system�[0m
�[32m13:12:28�[0m :: �[1;35m   Sourcer   �[0m :: �[1;33m DEBUG  �[0m :: �[1;37mFound valid FFmpeg executable: `ffmpeg`.�[0m
�[32m13:12:28�[0m :: �[1;35m   Sourcer   �[0m :: �[1;33m DEBUG  �[0m :: �[1;37mExtracting Metadata...�[0m
�[32m13:12:28�[0m :: �[1;35m   Sourcer   �[0m :: �[1;33m DEBUG  �[0m :: �[1;37mMetadata Extraction completed successfully!�[0m
�[32m13:12:28�[0m :: �[1;35m  FFdecoder  �[0m :: �[1;36m  INFO  �[0m :: �[1;37mUsing default `rgb24` pixel-format for this pipeline.�[0m
�[32m13:12:28�[0m :: �[1;35m  FFdecoder  �[0m :: �[1;36m  INFO  �[0m :: �[1;37mDefault source resolution will be used for this pipeline for defining output resolution.�[0m
�[32m13:12:28�[0m :: �[1;35m  FFdecoder  �[0m :: �[1;36m  INFO  �[0m :: �[1;37mDefault source framerate will be used for this pipeline for defining output framerate.�[0m
�[32m13:12:28�[0m :: �[1;35m  FFdecoder  �[0m :: �[1;31m�[47mCRITICAL�[0m :: �[1;37mActivating Video-Only Mode of Operation.�[0m
�[32m13:12:28�[0m :: �[1;35m  FFdecoder  �[0m :: �[1;33m DEBUG  �[0m :: �[1;37mExecuting FFmpeg command: `ffmpeg -vcodec hevc -i /Users/asia/Downloads/frames/HacksawRidge.2016.BluRay.1080p.x265.10bit.2Audio-MiniHD.mkv -pix_fmt rgb24 -s 1920x804 -framerate 23.98 -ss 00:30:00.00 -t 120s -r 0.5 -f rawvideo -`�[0m
ffmpeg version 6.0 Copyright (c) 2000-2023 the FFmpeg developers
  built with Apple clang version 14.0.0 (clang-1400.0.29.202)
  configuration: --prefix=/opt/homebrew/Cellar/ffmpeg/6.0 --enable-shared --enable-pthreads --enable-version3 --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libaribb24 --enable-libbluray --enable-libdav1d --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librist --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libsvtav1 --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libspeex --enable-libsoxr --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack --enable-videotoolbox --enable-neon
  libavutil      58.  2.100 / 58.  2.100
  libavcodec     60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  3.100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter     9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample   4. 10.100 /  4. 10.100
  libpostproc    57.  1.100 / 57.  1.100
[matroska,webm @ 0x148304130] Could not find codec parameters for stream 3 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size
Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options
[matroska,webm @ 0x148304130] Could not find codec parameters for stream 4 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size
Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options
[matroska,webm @ 0x148304130] Could not find codec parameters for stream 5 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size
Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options
[matroska,webm @ 0x148304130] Could not find codec parameters for stream 6 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size
Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options
[matroska,webm @ 0x148304130] Could not find codec parameters for stream 7 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size
Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options
Input #0, matroska,webm, from '/Users/asia/Downloads/frames/HacksawRidge.2016.BluRay.1080p.x265.10bit.2Audio-MiniHD.mkv':
  Metadata:
    encoder         : libebml v1.3.4 + libmatroska v1.4.5
    creation_time   : 2017-02-13T00:44:49.000000Z
  Duration: 02:19:19.49, start: 0.000000, bitrate: 6485 kb/s
  Chapters:
    Chapter #0:0: start 0.000000, end 558.765000
      Metadata:
        title           : Chapter 01
    Chapter #0:1: start 558.765000, end 1121.993000
      Metadata:
        title           : Chapter 02
    Chapter #0:2: start 1121.993000, end 1594.509000
      Metadata:
        title           : Chapter 03
    Chapter #0:3: start 1594.509000, end 1980.560000
      Metadata:
        title           : Chapter 04
    Chapter #0:4: start 1980.560000, end 2404.149000
      Metadata:
        title           : Chapter 05
    Chapter #0:5: start 2404.149000, end 3007.211000
      Metadata:
        title           : Chapter 06
    Chapter #0:6: start 3007.211000, end 3421.541000
      Metadata:
        title           : Chapter 07
    Chapter #0:7: start 3421.541000, end 3861.648000
      Metadata:
        title           : Chapter 08
    Chapter #0:8: start 3861.648000, end 4670.207000
      Metadata:
        title           : Chapter 09
    Chapter #0:9: start 4670.207000, end 5038.698000
      Metadata:
        title           : Chapter 10
    Chapter #0:10: start 5038.698000, end 5481.725000
      Metadata:
        title           : Chapter 11
    Chapter #0:11: start 5481.725000, end 5942.810000
      Metadata:
        title           : Chapter 12
    Chapter #0:12: start 5942.810000, end 6367.569000
      Metadata:
        title           : Chapter 13
    Chapter #0:13: start 6367.569000, end 7188.305000
      Metadata:
        title           : Chapter 14
    Chapter #0:14: start 7188.305000, end 7633.292000
      Metadata:
        title           : Chapter 15
    Chapter #0:15: start 7633.292000, end 8359.488000
      Metadata:
        title           : Chapter 16
  Stream #0:0: Video: hevc (Main 10), yuv420p10le(tv, unknown/bt709/unknown), 1920x804, SAR 1:1 DAR 160:67, 23.98 fps, 23.98 tbr, 1k tbn (default)
    Metadata:
      BPS             : 5486317
      BPS-eng         : 5486317
      DURATION        : 02:19:19.435000000
      DURATION-eng    : 02:19:19.435000000
      NUMBER_OF_FRAMES: 200426
      NUMBER_OF_FRAMES-eng: 200426
      NUMBER_OF_BYTES : 5732814506
      NUMBER_OF_BYTES-eng: 5732814506
      _STATISTICS_WRITING_APP: mkvmerge v9.7.1 ('Pandemonium') 64bit
      _STATISTICS_WRITING_APP-eng: mkvmerge v9.7.1 ('Pandemonium') 64bit
      _STATISTICS_WRITING_DATE_UTC: 2017-02-13 00:44:49
      _STATISTICS_WRITING_DATE_UTC-eng: 2017-02-13 00:44:49
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:1(eng): Audio: ac3, 48000 Hz, 5.1(side), fltp, 640 kb/s (default)
    Metadata:
      BPS             : 640000
      BPS-eng         : 640000
      DURATION        : 02:19:19.424000000
      DURATION-eng    : 02:19:19.424000000
      NUMBER_OF_FRAMES: 261232
      NUMBER_OF_FRAMES-eng: 261232
      NUMBER_OF_BYTES : 668753920
      NUMBER_OF_BYTES-eng: 668753920
      _STATISTICS_WRITING_APP: mkvmerge v9.7.1 ('Pandemonium') 64bit
      _STATISTICS_WRITING_APP-eng: mkvmerge v9.7.1 ('Pandemonium') 64bit
      _STATISTICS_WRITING_DATE_UTC: 2017-02-13 00:44:49
      _STATISTICS_WRITING_DATE_UTC-eng: 2017-02-13 00:44:49
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:2(chi): Audio: ac3, 48000 Hz, stereo, fltp, 256 kb/s
    Metadata:
      BPS             : 256000
      BPS-eng         : 256000
      DURATION        : 02:19:19.488000000
      DURATION-eng    : 02:19:19.488000000
      NUMBER_OF_FRAMES: 261234
      NUMBER_OF_FRAMES-eng: 261234
      NUMBER_OF_BYTES : 267503616
      NUMBER_OF_BYTES-eng: 267503616
      _STATISTICS_WRITING_APP: mkvmerge v9.7.1 ('Pandemonium') 64bit
      _STATISTICS_WRITING_APP-eng: mkvmerge v9.7.1 ('Pandemonium') 64bit
      _STATISTICS_WRITING_DATE_UTC: 2017-02-13 00:44:49
      _STATISTICS_WRITING_DATE_UTC-eng: 2017-02-13 00:44:49
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:3(eng): Subtitle: hdmv_pgs_subtitle (default)
    Metadata:
      BPS             : 14213
      BPS-eng         : 14213
      DURATION        : 02:09:49.127000000
      DURATION-eng    : 02:09:49.127000000
      NUMBER_OF_FRAMES: 3316
      NUMBER_OF_FRAMES-eng: 3316
      NUMBER_OF_BYTES : 13838703
      NUMBER_OF_BYTES-eng: 13838703
      _STATISTICS_WRITING_APP: mkvmerge v9.7.1 ('Pandemonium') 64bit
      _STATISTICS_WRITING_APP-eng: mkvmerge v9.7.1 ('Pandemonium') 64bit
      _STATISTICS_WRITING_DATE_UTC: 2017-02-13 00:44:49
      _STATISTICS_WRITING_DATE_UTC-eng: 2017-02-13 00:44:49
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:4(chi): Subtitle: hdmv_pgs_subtitle
    Metadata:
      BPS             : 23217
      BPS-eng         : 23217
      DURATION        : 02:13:03.100000000
      DURATION-eng    : 02:13:03.100000000
      NUMBER_OF_FRAMES: 3423
      NUMBER_OF_FRAMES-eng: 3423
      NUMBER_OF_BYTES : 23168356
      NUMBER_OF_BYTES-eng: 23168356
      _STATISTICS_WRITING_APP: mkvmerge v9.7.1 ('Pandemonium') 64bit
      _STATISTICS_WRITING_APP-eng: mkvmerge v9.7.1 ('Pandemonium') 64bit
      _STATISTICS_WRITING_DATE_UTC: 2017-02-13 00:44:49
      _STATISTICS_WRITING_DATE_UTC-eng: 2017-02-13 00:44:49
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:5(chi): Subtitle: hdmv_pgs_subtitle
    Metadata:
      BPS             : 23882
      BPS-eng         : 23882
      DURATION        : 02:13:03.100000000
      DURATION-eng    : 02:13:03.100000000
      NUMBER_OF_FRAMES: 3423
      NUMBER_OF_FRAMES-eng: 3423
      NUMBER_OF_BYTES : 23831726
      NUMBER_OF_BYTES-eng: 23831726
      _STATISTICS_WRITING_APP: mkvmerge v9.7.1 ('Pandemonium') 64bit
      _STATISTICS_WRITING_APP-eng: mkvmerge v9.7.1 ('Pandemonium') 64bit
      _STATISTICS_WRITING_DATE_UTC: 2017-02-13 00:44:49
      _STATISTICS_WRITING_DATE_UTC-eng: 2017-02-13 00:44:49
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:6(chi): Subtitle: hdmv_pgs_subtitle
    Metadata:
      BPS             : 22075
      BPS-eng         : 22075
      DURATION        : 02:13:03.100000000
      DURATION-eng    : 02:13:03.100000000
      NUMBER_OF_FRAMES: 3423
      NUMBER_OF_FRAMES-eng: 3423
      NUMBER_OF_BYTES : 22028598
      NUMBER_OF_BYTES-eng: 22028598
      _STATISTICS_WRITING_APP: mkvmerge v9.7.1 ('Pandemonium') 64bit
      _STATISTICS_WRITING_APP-eng: mkvmerge v9.7.1 ('Pandemonium') 64bit
      _STATISTICS_WRITING_DATE_UTC: 2017-02-13 00:44:49
      _STATISTICS_WRITING_DATE_UTC-eng: 2017-02-13 00:44:49
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
  Stream #0:7(chi): Subtitle: hdmv_pgs_subtitle
    Metadata:
      BPS             : 22543
      BPS-eng         : 22543
      DURATION        : 02:13:03.100000000
      DURATION-eng    : 02:13:03.100000000
      NUMBER_OF_FRAMES: 3423
      NUMBER_OF_FRAMES-eng: 3423
      NUMBER_OF_BYTES : 22495941
      NUMBER_OF_BYTES-eng: 22495941
      _STATISTICS_WRITING_APP: mkvmerge v9.7.1 ('Pandemonium') 64bit
      _STATISTICS_WRITING_APP-eng: mkvmerge v9.7.1 ('Pandemonium') 64bit
      _STATISTICS_WRITING_DATE_UTC: 2017-02-13 00:44:49
      _STATISTICS_WRITING_DATE_UTC-eng: 2017-02-13 00:44:49
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> rawvideo (native))
Press [q] to stop, [?] for help
[vost#0:0/rawvideo @ 0x1482058c0] Finishing stream without any data written to it.
Output #0, rawvideo, to 'pipe:':
  Metadata:
    encoder         : Lavf60.3.100
  Chapters:
    Chapter #0:0: start 0.000000, end 120.000000
      Metadata:
        title           : Chapter 04
  Stream #0:0: Video: rawvideo (RGB[24] / 0x18424752), rgb24, 1920x804 [SAR 1:1 DAR 160:67], q=2-31, 18524 kb/s, 0.50 fps, 0.50 tbn (default)
    Metadata:
      BPS             : 5486317
      BPS-eng         : 5486317
      DURATION        : 02:19:19.435000000
      DURATION-eng    : 02:19:19.435000000
      NUMBER_OF_FRAMES: 200426
      NUMBER_OF_FRAMES-eng: 200426
      NUMBER_OF_BYTES : 5732814506
      NUMBER_OF_BYTES-eng: 5732814506
      _STATISTICS_WRITING_APP: mkvmerge v9.7.1 ('Pandemonium') 64bit
      _STATISTICS_WRITING_APP-eng: mkvmerge v9.7.1 ('Pandemonium') 64bit
      _STATISTICS_WRITING_DATE_UTC: 2017-02-13 00:44:49
      _STATISTICS_WRITING_DATE_UTC-eng: 2017-02-13 00:44:49
      _STATISTICS_TAGS: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
      encoder         : Lavc60.3.100 rawvideo
frame=    0 fps=0.0 q=0.0 Lsize=       0kB time=-577014:32:22.77 bitrate=  -0.0kbits/s speed=N/A    
video:0kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Output file is empty, nothing was encoded (check -ss / -t / -frames parameters if used)
Exiting normally, received signal 15.
�[32m13:12:49�[0m :: �[1;35m  FFdecoder  �[0m :: �[1;33m DEBUG  �[0m :: �[1;37mTerminating FFdecoder Pipeline...�[0m
�[32m13:12:49�[0m :: �[1;35m  FFdecoder  �[0m :: �[1;36m  INFO  �[0m :: �[1;37mPipeline already terminated.�[0m
Processed 0 frames, skipped_by_similarity: 0 frames

Python Code(Optional)

No response

DeFFcode Version

0.2.5

Python version

3.11

Operating System version

macOS M1 ( Darwin Kernel Version 23.2.0: Wed Nov 15 21:53:18 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T6000 arm64)

Any other Relevant Information?

No response

The text was updated successfully, but these errors were encountered:

abhiTronix · 2024-04-05T12:22:03Z

@humpydonkey Thanks for the issue, I didn't noticed it until now. Thankfully it's a easy fix. :)

humpydonkey added Bug 🐞 Something's wrong with Deffcode APIs. Needs Triage 🧐 Assign priority to this issue. labels Mar 21, 2024

humpydonkey assigned abhiTronix Mar 21, 2024

humpydonkey changed the title ~~[Bug]: frame extraction by timestamp operation is very slow~~ [Bug]: frame extraction by timestamp operation can be very slow Mar 21, 2024

abhiTronix added Good Work 🌟 Good Issue/PR for newcomers to learn from. WIP 🏗️ Work in Progress Priority Normal ⚪ High Priority Issue/PR. and removed Needs Triage 🧐 Assign priority to this issue. labels Apr 5, 2024

abhiTronix added this to To do in Deffcode v0.2.6 via automation Apr 5, 2024

abhiTronix added this to the v0.2.6 milestone Apr 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: frame extraction by timestamp operation can be very slow #44

[Bug]: frame extraction by timestamp operation can be very slow #44

humpydonkey commented Mar 21, 2024

abhiTronix commented Apr 5, 2024

[Bug]: frame extraction by timestamp operation can be very slow #44

[Bug]: frame extraction by timestamp operation can be very slow #44

Comments

humpydonkey commented Mar 21, 2024

Description

Issue Checklist

Expected behaviour

Actual behaviour

Steps to reproduce

Terminal log output

Python Code(Optional)

DeFFcode Version

Python version

Operating System version

Any other Relevant Information?

abhiTronix commented Apr 5, 2024