custom data pretrain

[LiXinghui-666]

Hi, I would like to ask if I want to train this model on some of my own 3D models to achieve Multi-modal 3D Shape Retrieval task, what do I need to do with the original 3D data to provide the necessary data types for training the model, such as images, text, etc. Can you give me some suggestions or is it convenient to provide some scripts for data preprocessing?

[Tycho-Xue]

Hi @LiXinghui-666, if your own 3D models don't come with well-paired 3d-text data, then you might want to check "ULIP-2" which we released recently(arxiv link is in this repo), in "ULIP-2", only the 3D models is required.
you basically need to do:

1. extract point cloud (if you are retrieving by point cloud)
2. render the 3D models (.obj file or whatever format you have) to holistic views of images.
3. use image captioning models (we used BLIP-2 in the "ULIP-2") to caption each rendered image from step-2
4. up until now, you have got tri-modal datasets, use ULIP's framework to train the 3D encoder.
5. do the retrieval

Let me know if this makes sense or if you need more help.
ULIP-2 is the framework which we are enabling multimodal pre-training where only 3D models themself are needed

[LiXinghui-666]

Thank you for your patient answer! I want to know if the ULIP-2 model will be open source?

[Tycho-Xue]

Yes, I should be able to upload it tomorrow

…

On Tue, Jun 6, 2023 at 10:39 PM LiX ***@***.***> wrote: Thank you for your patient answer! I want to know if the ULIP-2 model will be open source? — Reply to this email directly, view it on GitHub <#33 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHTPKPZFJOSMKLSHEAZPD2LXKAHY7ANCNFSM6AAAAAAY5I44SY> . You are receiving this because you commented.Message ID: ***@***.***>

[LiXinghui-666]

Oh! Looking forward to your released codes which can also include the code of the multimodal retrieval task. And I expect ulip2 pre-trained models can also be available for download. I am amazing about the model effect in the paper and can't wait to try it!