added better guidance for if deprecated tokenizer path fails

[wesleytruong]

Adds a check to see if the old tokenizer path is being used when tokenizer path fails. This way it can provide guidance to people to update to the supported hf_assets_path and download_hf_assets.py script

[XilunWu]

LGTM but i have some suggestions.

[XilunWu]

I suggest we add link https://github.com/pytorch/torchtitan/tree/main/torchtitan/models/deepseek_v3#download-tokenizer for better user guidance.

[XilunWu]

what about adding more info about path deprecation:
"detected ... path that's deprecated in #1526"

[XilunWu]

I think we don't need to change the check here because the logic was:

• check "tokenizer_path" folder for available tokenizer files.

While the new logic we add is to remind users of the deprecation of old tokenizer path. Since we have verified the correctness of "tokenizer_path", there's no need to perform it again -- if the execution goes to the else branch, that simply means the specified tokenizer file is not available and the existing information is detailed enough.

[fegin]

If the logic is not required as @XilunWu mentioned, please removed it. If the logic is required, this if/else plus the error messages look almost identical to the error handling block above, please consider to make it as a function.

[wesleytruong]

I think we don't need to change the check here because the logic was:

• check "tokenizer_path" folder for available tokenizer files.

While the new logic we add is to remind users of the deprecation of old tokenizer path. Since we have verified the correctness of "tokenizer_path", there's no need to perform it again -- if the execution goes to the else branch, that simply means the specified tokenizer file is not available and the existing information is detailed enough.

Ok I think I'll remove the second check. I added it because I wanted to cover the use case of people who may have this path due to using legacy script but are missing the tokenizer file, but I realize people will probably either not have this path at all or have the path with the files as well.

[fegin]

Stamp to unblock, please address the comments before landing.

[XilunWu]

any reason for the extra space?

[ruisizhang123]

Hi, found this pr after failing to load tokenizer in deepseekv3 due to the assertion error in tokenizer_path = "./assets/tokenizer/deepseek-moe-16b-base". (Link)

Wonder if there is a reason for not updating it as assets/hf or not updating hf_assets_path in deepseekv3 config here?

[wesleytruong]

Hi, found this pr after failing to load tokenizer in deepseekv3 due to the assertion error in tokenizer_path = "./assets/tokenizer/deepseek-moe-16b-base". (Link)

Wonder if there is a reason for not updating it as assets/hf or not updating hf_assets_path in deepseekv3 config here?

Sorry for the inconvenience, I've been waiting until #1582 lands since the current hf_asset_path currently isn't well supported for special multimodal nested repositories such as in FLUX experiment without specifying initial_load_path separately from hf_assets_path when using initial_load_in_hf.

You bring up a good point though, I'll change the toml files now for non-flux models.