You can train a tokenizer on old data just like you can train a model on old dat... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		clickety_clack 42 days ago \| parent \| context \| favorite \| on: Changes in the system prompt between Claude Opus 4... You can train a tokenizer on old data just like you can train a model on old data.

wongarsu 42 days ago [–]

But you can't use an old model with a new tokenizer. Changing the tokenizer implies you trained the model from scratch

dannyw 41 days ago | [–]

A little bit of post-training will fix that. Folks on /r/LocalLLaMa have been making effective finetunes with diff. tokenizers for years.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact