[VitTrack Evaluation] OTB100 Dataset Implementation #245
Conversation
fengyuentau
added
the
evaluation
|
@lpylpy0514 Could you verify whether this accuracy test results are close to yours? |
|
I tested VitTrack based on my pytorch code and got 56.98 AUC on OTB-100. |
Thank you! @ryan1288 Could you also provide AUC in the script? |
|
@fengyuentau In order to figure out what is different between the two methods, I need to be able to run the evaluation script from @lpylpy0514 's repository. Would you be able to help with this? @lpylpy0514 |
|
Try this. |
|
@fengyuentau @lpylpy0514
|
Good progress 👍 Let's see whether we are comparing two differnet models or two identical models. |
|
Two models are almost same, but I found a bug in my post-process. The hanning window implementation in openCV is not same as my pytorch code, and this may cause some difference. It seems that the implementation in openCV is more accurate based on @ryan1288 's result, but the result may also differ in different dataset. I got 57.55 AUC with the same post-process in openCV and OstrackDist_ep0300.pth.tar. |
Thank you for verifying all these details👍 I think we should make sure that the implementation of algorithm in OpenCV is correct, so feel free to create pull request for the bug fix. |
|
With patch opencv/opencv#25264, I get AUC 0.579 using this evaluation script, which is still different from 57.55 (0.5755) from @lpylpy0514 's result. @ryan1288 Could you compare the difference between your script and the one used by @lpylpy0514 ? |
|
@fengyuentau Yup, will do. I'll both update the OTB dataset process to not rename/remove & find any differences between the processes in the next day or two. |
|
The data I mentioned above is generated without post-process. It's the difference between pytorch and onnx. I think small difference between them is reasonable. |
|
@lpylpy0514 Yes I think we're on the same page. The model difference is minimal, so I'll be taking a look at the post-processing to see where the difference is in the next few days. |
There should be very minimal difference between pytorch and onnx. I don't quite understand |
|
Yes, the data above is part of conf_map. |
…. Removed all moving or renaming or files and directories
|
@fengyuentau I've made some changes to:
Although the outputs still do not match for me, I suspect it's because I don't have the patch from opencv/opencv#25264. I'm to try it with his patch now, as I suspect that fixing the testing dataset to match @lpylpy0514 's method may possibly close the gap.
Side note: I've been finishing up some paper rebuttals, so I couldn't update this earlier. Regarding GSoC, I put in a post within GSoC Google Group but I don't think anyone has gotten a response yet. Please let me know if mentors generally want a longer, detailed, and more specific proposal. I'd love to work on similar (or different!) things in the summer as my research is ending so please keep me updated 😄 Thanks! |
|
I rebuilt the most recent version of OpenCV (with the patch) and am getting 0.586. Could you please try it? @fengyuentau Perhaps it's a setup issue in how I rebuilt OpenCV for the patch. Thanks! |
@ryan1288 I am still getting the quoted results (without OTB100.json) with your new commits and latest OpenCV which has the patch merged. What I did additionally is only changing the comma-separated line.
Do we need to do anything regarding this? If no, I will rezip the files.
We are quite busy with OpenCV 5 release so few of us can respond to GSoC mail group. You can take a look at |
There was a problem hiding this comment.
As @lpylpy0514 says that minor AUC difference is accepted, we can merge this PR after the following comments are resolved.
Other than the one unusual line in Regarding GSoC OpenCV5, I have more experience in C++ than Python, despite my previous contributions to OpenCV Zoo being primarily in Python. While I acknowledge that my C++ proficiency might not match the standards of a complex open-source project like OpenCV, I am enthusiastic about taking on a well-defined project and am committed to learning whatever is necessary to make meaningful contributions. For instance, as a focused effort within the imgproc enhancements outlined in issue #25001, I could tackle the implementation of a refined version of |
tools/eval/datasets/otb100.py
Outdated
| with open(result_path, 'w') as f: | ||
| for bbox in pred_bboxes: | ||
| f.write(','.join(map(str, bbox)) + '\n') |
There was a problem hiding this comment.
We can drop result saving for now as they are not needed after evaluation is finished and we do not load them for the second time evaluation.
There was a problem hiding this comment.
Replaced with a global dict. If this isn't preferred, I can find another way tomorrow. 👍
There was a problem hiding this comment.
Seems like it is not set correctly:
python eval.py -m vittrack -d otb100 -dr ../../benchmark/data/
Evaluating: 100%|█████████████████████████████████████████████████| 100/100 [09:16<00:00, 5.57s/it]
No prediction found for video Basketball
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/Workspace/miniconda3/lib/python3.10/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/Workspace/fytao/opencv_zoo/tools/eval/datasets/otb100.py", line 64, in evaluate
evaluation_ret[video.name] = success_overlap(gt_traj, tracker_traj, n_frame)
File "/Workspace/fytao/opencv_zoo/tools/eval/datasets/otb100.py", line 26, in success_overlap
iou = overlap_ratio(gt_bb[mask], result_bb[mask])
IndexError: too many indices for array: array is 0-dimensional, but 1 were indexed
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Workspace/fytao/opencv_zoo/tools/eval/eval.py", line 156, in <module>
main(args)
File "/Workspace/fytao/opencv_zoo/tools/eval/eval.py", line 153, in main
dataset.print_result()
File "/Workspace/fytao/opencv_zoo/tools/eval/datasets/otb100.py", line 289, in print_result
for ret in pool.imap_unordered(benchmark.evaluate, [metric], 1):
File "/Workspace/miniconda3/lib/python3.10/multiprocessing/pool.py", line 873, in next
raise value
IndexError: too many indices for array: array is 0-dimensional, but 1 were indexed
There was a problem hiding this comment.
Have you pulled recently? With the current branch, I am able to run the full evaluation with python3 eval.py -m vittrack -d otb100 -dr OTB100. I have no extra files. Worth noting that I do get different results than you, presumably because I don't have the patch properly applied locally.
There was a problem hiding this comment.
Which opencv version are you using? I can confirm that both my latest built-from-source version and latest release of opencv do not work. Checking back to the last commit works. Did you do some modifications to the dataset or the opencv code?
There was a problem hiding this comment.
Using the dataset zip that you uploaded. I was able to run it using version 4.9.0.
I am able to:
- Create a new folder
git clone git@github.com:ryan1288/opencv_zoo.gitgit checkout ryan-vittracker_eval- Move the new downloadable dataset into the
tools/evalfolder and unzip cd tools/evalpython3 eval.py -m vittrack -d otb100 -dr OTB100
And it runs without problems.
I'll check my OpenCV installation to ensure there aren't discrepancies tomorrow. You're significantly more likely to have set it up correctly than me 👍
If you are interested in any project for OpenCV 5, please do leave comment in the issue first and have discussion before contribution, so that your time will not be wasted :) |
|
I re-zipped the dataset with correction on the missing comma: https://drive.google.com/file/d/1TaGOu7FfTKRM57pgRsdFCa-BYHmaxJI1/view?usp=sharing. |
|
I'm finishing up a paper rebuttal but I'll take a look at this soon next week. I'm sure we can figure out why there's a discrepancy between our builds! |
|
Hey @fengyuentau, finally got around to it. I completely uninstalled all opencv libraries from
I have strong reason to believe that I set up my OpenCV correctly because I was able to replicate your 0.579 AUC (and other two metrics) as shown, indicating that I correctly have @lpylpy0514's patch fix from OpenCV
Could you please check if this process works correctly? I haven't changed any code since my last commit but it seems to work as expected on my local machine. Thank you @fengyuentau. |
|
Still does not work on my side. Could you make sure that you are on the lastest commit? |
|
Looks like it is caused by no prediction for class Basketball. |
| from tqdm import tqdm | ||
| from multiprocessing import Pool, cpu_count | ||
|
|
||
| PRED_BBOXES_DICT = {} |
There was a problem hiding this comment.
I suspect that the problem is this global variable. Please keep detection results in each corresponding class instead.
This PR implements the evaluation script for VitTrack using the OTB100 dataset, as part of an issue raised in #119. Reference was made to #205 during the development process.
eval.py: Referenced VitTrack's onnx model and OTB as the tracking evaluation dataset.__init__.py: Added OTB dataset.README.md: Updated with instructions for dataset preparation and evaluation. Simplification from Eval tracker #205 ensures complete automation of dataset preparation post-download.Testing:
python eval.py -m vittrack -d otb -dr /path/to/otbtools/evaldirectory,python eval.py -m vittrack -d otb -dr .Both procedures yield identical results regardless of the presence of the OTB100.zip file, the correct placement of OTB.json, or the preprocessing status of the folders.
