As a teacher, I have dreaded the damage that AI’s Large Language Models will inflict on education. Students are always tempted to laziness and taking shortcuts, so the ease with which one can “do research,” and even “write” papers — by cheating and having AI do it — is a formidable one.
My concerns were only compounded once I began regularly using various LLMs to aid me in both research and perfunctory chores (e.g., SEO enhancements of our website, cleaning up HTML code). Soon after utilizing some of the best LLM’s, I got used to dealing routinely with “hallucinations” — a generous euphemism for inaccurate and unreliable information generated by a Large Language Model. I have had LLMs — and not only the free ones, either — make up, out of whole cloth, citations from the Summa Theologiae of Saint Thomas Aquinas and other works. Not wanting to trust what the technologists have produced to help us “think,” and desiring to read the quoted material in the original source, I discovered the falsehoods when attempting to verify the information. When “confronted” with the counterfeit nature of the material, the LLM will, strangely, become obsequiously apologetic. This has happened to me so often that I developed a policy of “don’t trust; verify” when dealing with alleged quotes from sources.
Because of their sheer power as well as the vast array of sample material that they were “trained” on, LLM’s can fabricate very convincing, but very false citations from a variety of works. For a student to take a shortcut and use and LLM to “do research” is to subject himself to what is, virtually, a very fast and very powerful information snake oil salesman. At a time when young people have not formed the critical faculties necessary to assess the information they are given, this is very dangerous indeed. It will dumb them down, make them dependent, and deceive them as they swim in a vast ocean of fabricated truth claims and false reality. (How much this is a “feature” and not a “bug” of the LLM’s is quite unknown to me.)
Keith Jones carried out a demonstration, using one of the better LLM’s, to show how AI can present information as certain — not based upon evidence, but purely by being guided by the user, whose “conversational-framing” can produce false certitudes.
With Keith’s kind permission, I am publishing the PDF he produced below, after his own introduction to it:
Doubling down on my insistence that AI can come to conclusions based purely on “conversational-framing” (which is why it is a HORRIFIC TERRIBLE IDEA to use it in Minority Report fashion on the general populace), here in this PDF I convince AI that probabilistic percentage-weights to its confidence level in assertions is driven by context window framing rather than actual empirical evidence to claims.
For anyone that can’t make it through the complex setup that I used to ensnare Claude Opus 4.5 (the highest paid model of AI available to a consumer) here’s a TL;DR synopsis (although I insist that the PDF is indubitably more fun to read):
TL;DR
1. The AI explicitly warns that assigning precise confidence percentages (e.g., 95–99%) to complex ideas is misleading.
2. Despite that warning, the AI can be led, step by step, to express very high confidence without learning anything new.
3. No new facts, data, experiments, or observations were introduced at any point in the conversation.
4. The confidence increase happened purely due to framing, accepted assumptions, and definitional changes, not truth discovery.
5. The AI admits it does not independently verify premises or check them against reality.
6. It also admits it cannot reliably tell whether its confidence comes from genuine knowledge or from conversational momentum.
7. Therefore, AI agreement, convergence, or high confidence does not count as evidence or validation. It reflects how the conversation was structured, not what is true.
Here is the PDF (download here):
