Towards Scene Graph Anticipation

Rohith Peddi, Saksham Singh, Saurabh, Parag Singla, Vibhav Gogate

(This page is under continuous update)

---

UPDATE

Jun 2024: Release of pre-trained models and updated code.

Mar 2024: Released code for the paper

---

TASK PICTURE

---

TECHNICAL APPROACH

---

ACKNOWLEDGEMENTS

This code is based on the following awesome repositories. We thank all the authors for releasing their code.

1. STTran
2. DSG-DETR
3. Tempura
4. TorchDiffEq
5. TorchDyn

---

HOW TO RUN THE CODE

1. To run the baseline transformer-based methods, please follow the instructions on STTran and DSG Detr
2. To run SceneSayer models, you need to use remove_object_detector.py code to store features and use them as input to the SDE and ODE models.

AFTER FEATURE EXTRACTION

Build draw_rectangles modules

cd lib/draw_rectangles

Remove any previous builds

rm -rf build/
rm -rf *.so
rm -rf *.c
rm -rf *.pyd

Build the module

python setup.py build_ext --inplace
cd ..

Add the path to the current directory to the PYTHONPATH

conda develop draw_rectangles/

Install required libraries

conda create -n sga python=3.7 pip

conda activate sga

pip install torch==1.10.1+cu111 torchvision==0.11.2+cu111 torchaudio==0.10.1 -f https://download.pytorch.org/whl/cu111/torch_stable.html

pip install -r sga_requirements.txt

Scene Sayer

Scene Sayer

conda create -n sgg python=3.7 pip
conda activate sgg

pip install torch==1.10.1+cu111 torchvision==0.11.2+cu111 torchaudio==0.10.1 -f https://download.pytorch.org/whl/cu111/torch_stable.html
pip install -r requirements.txt

Dataset Preparation

Estimated time: 10 hours

Follow the instructions from here

Download Charades videos data/ag/videos

Download all action genome annotations data/ag/annotations

Dump all frames data/ag/frames

Change the corresponding data file paths in datasets/action_genome/tools/dump_frames.py

Download object_bbox_and_relationship_filtersmall.pkl from here and place it in the data loader folder

Setup

Build draw_rectangles modules

cd lib/draw_rectangles

Remove any previous builds

rm -rf build/
rm -rf *.so
rm -rf *.c
rm -rf *.pyd

Build the module

python setup.py build_ext --inplace
cd ..

Add the path to the current directory to the PYTHONPATH

conda develop draw_rectangles/

Build bbox modules

cd fpn/box_intersections_cpu

Remove any previous builds

rm -rf build/
rm -rf *.so
rm -rf *.c
rm -rf *.pyd

Build the module

python setup.py build_ext --inplace
cd ..

Add the path to the current directory to the PYTHONPATH

conda develop fpn/box_intersections_cpu/

fasterRCNN model

Remove any previous builds

cd fastRCNN/lib
rm -rf build/

Change the folder paths in 'fasterRCNN/lib/faster_rcnn.egg.info/SOURCES.txt' to the current directory

python setup.py build develop

If there are any errors, check gcc version Works for 9.x.x

Follow this for changing gcc version

Download pretrained fasterRCNN model here and place in fasterRCNN/models/

Citation

@misc{peddi2024scene,
      title={Towards Scene Graph Anticipation}, 
      author={Rohith Peddi and Saksham Singh and Saurabh and Parag Singla and Vibhav Gogate},
      year={2024},
      eprint={2403.04899},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}