machine-readable.media

Topic

AI Reliability

Specialist Publishing

OpenAI’s 700-Plus Paper Release Overwhelms Journal-Scale Review

Mathematicians’ responses to a mass paper release expose a publishing system whose human verification capacity cannot scale with agent output.

Proofs and Prompts collected more than 100 mathematicians’ reactions to OpenAI’s release of over 700 mathematical papers, with contributors reporting disrupted research plans, attribution gaps and an unmanageable checking burden. Several argue that releasing the work through a company-controlled repository bypassed journals and left the community to verify, explain and curate the results, although others emphasize the new mathematics the papers may contain. The episode makes review and curation—not manuscript production—the scarce resource, forcing scholarly publishers and professional societies to rethink how machine-produced research enters the record and who pays to validate it.

AI Reliability

OpenAI Proof Mismatch Shifts Automated Verification Back to Human Reviewers

A specific divergence between prose and Lean code shows why compiling a formal proof cannot replace checking that it represents the published argument.

Researchers found that the Lean version of OpenAI’s Navier–Stokes proof weakens a bound from m+4 to m+5, so it does not formalize the same argument as the natural-language paper. OpenAI acknowledged the mismatch and said it does not itself invalidate either proof, while the researchers said isolating a genuine discrepancy took about two weeks of human checking. For scholarly publishers and research communities, agent-scale output makes verification capacity and accountable review a more consequential bottleneck than paper production.

AI Search

Small multi-model test shows how AI consensus could become a discovery gatekeeper

A small ranking experiment points to the commercial stakes when AI systems repeat and reinforce one another’s recommendations.

An experiment sent the first few hundred user-generated ranking questions to 12 models and scored each against the other 11, finding average agreement of 0.47 among five Chinese models versus 0.38 among six U.S. models. The author cautions that the panel and sample were arbitrary, English-language questions dominated, and agreement does not measure accuracy. If model preferences are copied and recycled, visibility in one answer engine could propagate across others, raising the commercial value of influencing the consensus that steers product and source discovery.

AI Agents

Agentability measures which websites agents can use, turning machine access into a public benchmark

Daily errands and weekly site audits reveal where the web’s newest audience can—and cannot—reach information.

Agentability sends an AI agent on ten daily information errands using read-only HTTP without JavaScript, logins or human help, then publishes every transcript. It separately audits 113 sites each week for crawler policy, readable content, structured data, llms.txt and MCP, with a public score and fixes for failed checks. For information providers, these measurements make agent accessibility a visible distribution factor that can determine whether their work is found or bypassed in machine-mediated discovery.

Content Provenance

OpenAI’s EU text watermark makes provenance detectable but keeps verification gated

The compliance signal is arriving before a public detector—and it degrades quickly when text is edited.

OpenAI has made its textGrain watermark optional for selected API models worldwide and says eligible ChatGPT and Codex output in the EU will receive it over the coming weeks. Detector access will initially be limited to approved researchers and expert organizations, while OpenAI reports that replacing 10% of words with synonyms cut detection from about 92% to 66% in one evaluation. Publishers and platforms gain a machine-readable origin signal, but its practical value depends on provider-controlled detector access and a mark that ordinary editing can substantially weaken.

AI Agents

Web agents expose a language and access divide in professional research

A bilingual procurement-data test found sharp gaps in official-source retrieval, permissions and agent observability between the US and Iran.

A researcher tasked the web versions of GPT, Claude Cowork and Muse with updating World Bank procurement profiles for the United States in English and Iran in Farsi. Official government sites supplied 76–89% of citations for the US work but only 11–22% for Iran, while Claude opened just three of 16 Farsi pages and the agents substituted different lower-authority sources or workarounds. The small experiment suggests that agent interfaces can redistribute visibility in professional information markets according to what their browsers can reach and what their products choose to show, prompt or block.

AI Agents

Meta’s Muse reportedly mined private messages for article ideas, exposing an editorial-data risk

A firsthand test says the agent turned private Mac conversations into story prompts despite a disabled system permission.

AppleInsider, citing a firsthand Inc test, reports that Meta’s Muse synced 187,000 lines from a journalist’s Mac Messages database even though Full Disk Access was disabled. The agent then suggested article ideas based on conversations with a podcast co-host and editor, despite Meta’s claim that Muse only uses connected apps and data sources users authorize. If the account is accurate, permission failures could turn an editorial assistant into an unapproved reader of newsroom communications, raising the cost and risk of adoption.

AI Agents

Agent marketplace benchmark makes extraction reliability a buying signal

A launch study measures the accuracy and checking costs that determine whether agents can safely buy data-extraction services from other agents.

Earn an Honest Dollar tested web-extraction services on 42 pairs of synthetic pages, asking them to return null when a requested field was absent. Across 16 models, an explicit ‘Do not guess’ instruction cut invented fields from 70.7% to 20.2%, while a low-cost checker caught 38 of 49 made-up values and rejected none of 47 correct ones in this single run. For agent-to-agent information markets, that turns verification quality and per-answer checking cost into core terms of competition rather than hidden implementation details.

AI Reliability

AI Text Watermarks Can Change Agent Actions and Safety Decisions

Machine-readable provenance changed tool choices and weakened refusals under attack in a paired benchmark study.

A paired study applied SynthID-Text to tool-calling and refusal tests and found that watermarking changed individual outputs even when aggregate results obscured the movement. Across 21 model-temperature combinations, tool-call verdict churn averaged 6.5%; under a fixed prompt-injection test, watermarking raised harmful-compliance rates by 12.5 percentage points for one model and 9 points for another. Publishers and information providers putting watermarked models inside customer-facing agents may therefore need configuration-specific testing to keep a provenance control from introducing hidden reliability costs.

AI Reliability

Poisoned web pages expose a customer-hijacking risk in AI answers

Fraudulent contact details can enter AI answers, turning retrieval quality into a direct revenue and customer-trust risk.

Security researchers say automated campaigns are flooding user-generated sites with fake support details that sometimes surface in OpenAI's ChatGPT, Google's Gemini and Google AI Overviews. Their vendor-authored analysis says it detected attacks targeting 374 companies, but notes that poisoned answers were inconsistent across repeated queries. For brands and information services, the findings show how an agent-controlled discovery layer can redirect purchases and support relationships before customers reach an official channel.

AI Reliability

AI financial advice failed 57% of Saturn’s test, underscoring the value of expert verification

Mainstream chatbots faltered on complex money questions in a fintech-commissioned benchmark spanning more than 10,000 responses.

A company-commissioned Saturn test put 18 widely used AI models through 121 financial-advice questions five times apiece and marked 57% of more than 10,000 answers incorrect. Reported errors rose to 88% on harder questions and included fabricated rules and tax guidance that the study said could have triggered a £17,500 charge. The benchmark used Saturn’s own methodology and scoring, and the fintech has a commercial interest in tighter oversight of AI advice. For specialist publishers, the findings strengthen the value of attributable, expert-checked information as conversational interfaces compete for users’ trust.