Fix rope scaling factor

[abuelnasr0]

I think the use of scaling_factor is wrong in RotaryEmbedding layer. It is used to scale the positions not the frequencies.

References:

• The original idea of rope scaling: https://kaiokendev.github.io/til#extending-context-to-8k
• hf llama implementation: https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py#L108
• Meta paper: EXTENDING CONTEXT WINDOW OF LARGE LANGUAGE MODELS VIA POSITION INTERPOLATION

[mattdangerw]

This looks good! Can we add some tests?

[mattdangerw]

@abuelnasr0 also apologies, we just changed our entire directory structure in #1608

(Hopefully for good reason, we want to allow pip install -e . and pip install git+https:// while still keeping our explicit API surface)

But it does mean everything will need an annoying merge/rebase. If it'd help for me to do any of those and push to this branch just lmk!

[abuelnasr0]

Can we add some tests?

I can add tests, but on sunday. sorry for that, but I will be AFK until then.

[mattdangerw]

I can add tests, but on sunday. sorry for that, but I will be AFK until then.

No rush at all! And thanks so much for all the major contributions to the library :)

I am just getting back from vacation myself, slowly catching up on all the review.

[mattdangerw]

LGTM! If tests pass lets ship it!

[abuelnasr0]

thanks so much for all the major contributions to the library :)

You're welcome. I am trying to give back to the community as much as I can. And actually contributing to the library helped me to improve, I am learning new things with each PR. Thank you & other authors for creating the library. and thank you for all your reviews, they were really helpful to me.

[abuelnasr0]

https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py#L96-L174

checkout these lines. what is implemented in this PR is LlamaLinearScalingRotaryEmbedding. and I think it was implemented in the main layer at first, but they decided to move it to a new layer. there's also LlamaDynamicNTKScalingRotaryEmbedding that uses scaling_factor in another way. but I think LlamaLinearScalingRotaryEmbedding is more popular.