Are there any good resources related to expanding context windows, or even just ... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		apstls on Sept 13, 2023 \| parent \| context \| favorite \| on: Fine-tune your own Llama 2 to replace GPT-3.5/4 Are there any good resources related to expanding context windows, or even just the mechanics of how they actually work as properties of a model?

ttt3ts on Sept 13, 2023 [–]

Lots. LLAMA 2 was trained on 4K context windows but can run on arbitrary length just the results become garbage as you go longer.

I refer you to https://blog.gopenai.com/how-to-speed-up-llms-and-use-100k-c... for an "easy" to digest summary

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact