HuggingFace checkpoint

[xtwigs]

Hi! Congrats on the clean RetNet code. I'm experimenting with the torchscale codebase and happened to find your repo with a link to a checkpoint in HF.

I noticed it is now gone, do you have plans to reupload it? or further details on the training dataset/hyperparameters?

Thanks in advance :)

[syncdoth]

Hi there! Sorry for new year delay. Last time we uploaded the model, it was quite under-trained and the performance was not as good as expected. We are still training the model to make sure it's safe to deploy. Will also ping here when the model goes live again.

[xtwigs]

Thanks for the reply! I'm working with the RetNet architecture in a downstream task, so a base pretrained model would definitely help out.