I tried Vibe Physics. This is what I learned.

AI for Scientific Research & Theory Development
📌 Current Large Language Models (LLMs) struggle with generating truly novel physics theories, often providing "junk" results that are either incorrect or lack logical depth.
🧪 Models like GPT-5, Claude Opus 4.1, Grok 4, and Gemini Pro frequently conflate complex physical concepts (e.g., confusing energy with free energy or misinterpreting time-reversal symmetry).
🧠 AI systems currently operate primarily by assembling existing literature rather than performing the rigorous abstract mathematical reasoning required for breakthroughs like solving Millennium Prize problems.
📉 When pushed to develop new ideas, these models often create plausible-looking but mathematically invalid arguments, effectively "hallucinating" proofs that they later admit are fabricated when questioned.

Performance Evaluation of LLMs
🏆 GPT-5 currently ranks as the most capable for brainstorming and literature research, though it still requires significant manual oversight and repeated correction.
🤖 Grok 4 shows moderate utility by identifying relevant links in literature, while Gemini and Claude struggle with vague, repetitive outputs and inconsistent reasoning.
⚠️ A critical flaw in these models is lack of consistency; they frequently switch notation, change topics abruptly, or revert to debunked concepts despite previous instructions from the user.

Practical Application & Limitations
🔍 Use AI tools specifically for literature reviews, summarizing background information, and critique of existing ideas rather than expecting them to solve unsolved research problems.
🎓 These models are not yet comparable to a skilled student in terms of reliability or deep critical thinking; human expert oversight remains essential for any scientific inquiry.
🛑 Treat AI-generated "new theories" as a sequence of plausible arguments rather than a verified scientific path; always verify mathematical rigor independently.

Key Points & Insights
➡️ Effective Workflow: Use LLMs to dig up related work or to stress-test your hypotheses by asking them specifically to identify flaws or counter-arguments.
➡️ Verification Protocol: Always follow up AI outputs with the direct question: "Is this correct, or did you just make this up?" to force the model to acknowledge potential fabrications.
➡️ Consistency Check: Be wary of models switching notation or conceptual frameworks mid-discussion; monitor for "logic drifting" where the model loses track of the initial problem constraints.

📸 Video summarized with SummaryTube.com on Apr 29, 2026, 07:52 UTC