1/ Launching Enchanted ✨ Every human deserves a truly personal AI. Not one controlled by a big lab. Not one that hoards your data. Use Enchanted.

Haven't tweeted a lot in these last two months as I spent time learning and experimenting with various RL techniques. Excited to share some WIP soon: 1. Compute-optimal recipe for GRPO training 2. RL-powered tool to enhance privacy in LLM interactions The experiments have been promising 👀

1/ Today we announce Pleiades, a series of epigenetic foundation models (90M→7B params) trained on 1.9T tokens of human methylation & genomic data. Pleiades accurately models epigenetics for genomic track prediction, generation & neurodegenerative disease detection from cfDNA, outperforming previous pure DNA baselines.

What is the role of LLM memory/RAG in the short to medium term given in the long term context windows might be extremely large? Got some insights from discussing LLM context windows with someone on the DeepMind team. Working on Gemma, they found they could stretch context length but hit quality issues - tokens at beginning and end get retrieved well, but middle ones get lost in the attention mechanism. Their interesting take: they initially thought longer context would just solve everything, but deeper analysis showed that with fixed parameter count, quality doesn't come for free. Despite pushing context lengths further, they still see RAG approaches as necessary for the near future (6-12 months) because of these attention quality challenges. For now, the retrieval problem isn't fully solved just by making contexts longer. Besides, filling in the entire context window for high-quality, long-context models is ~$1 per call today.