Can shopping agents verify the total and refund terms on merchant sites?
A company-authored pilot found many merchant pages readable but too incomplete for a hard-budget agent to decide safely. The benchmark is unvalidated, commercially interested and does not measure completed transactions or disputes.
The short version
3 key points
- 1.In a one-time census of 555 publicly named processor clients, 105 home pages were unreadable to both a declared HTTP bot and a declared headless browser from one Canadian network location.
- 2.In a deterministic assessment of 330 mostly North American retailers, only 115 had enough evidence coverage to be ranked under at least one profile; the all-in total was decidable at only 18 merchants under each profile.
- 3.The work does not identify merchants, validate its weights, observe checkout or transactions, or show that gaps cause losses. The authors' company develops the framework being assessed.
The answer: access was often possible, but decisive terms were missing
Many merchant home pages could be read by declared automated profiles, yet the pages often did not expose enough information for an agent to prove that a purchase met a person's budget and refund conditions. In the largest detailed sample, uncertainty about tax, shipping and return terms mattered more than a site's overall technical score.
Of 330 merchants assessed with deterministic rules, only 115 had sufficient evidence coverage to be ranked under at least one of two declared profiles. The all-in total was decidable at just 18 merchants under each profile. A page that defers tax or shipping to checkout may serve a human shopper but leaves a hard-budget agent unable to certify the mandate in advance.
The study is a pilot, not a market ranking. It did not place orders, inspect transaction outcomes or establish that incomplete terms cause disputes. The benchmark's weights and clearing bands are unvalidated, and the authors work for the company developing the RAILS framework and merchant-assessment concept.[1]
A benchmark built around get in, decide and prove
The proposed unit is a merchant, a buyer mandate and an agent profile—not a website alone. The same pages may answer one instruction but leave another unresolved. Each observation records route, locale, currency, destination, login state, page sample, time and scanner version.
Get in asks whether the profile can reach an offer and its terms through a route the merchant permits. Decide asks whether the merchant's own statements determine that the offer satisfies the person's constraints. Prove classifies what evidence could later show the terms the agent accepted.
The paper uses a three-way state for each requirement: determinate, indeterminate or unknown. Scores carry evidence coverage and upper and lower bounds so a high score cannot hide missing pages. Access is reported separately for a declared HTTP client, a declared headless browser and, in four full assessments, a user-present browser.[1]
What the 555-site access census measured
The authors assembled 555 commercial domains from merchants that two payment processors had publicly named as clients by 29 September. The list was not random and likely favoured large brands. Shared hosts were merged and two domains that did not resolve were removed.
On 30 September, each site received about five requests from one network location in Ontario: robots.txt, the home page and two possible agent-protocol paths from a declared HTTP client, plus the home page from a declared headless browser. The probes identified themselves, respected robots rules and did not request cart, checkout, booking or login pages.
The HTTP profile read 418 home pages, or 75.3%; the headless profile read 423, or 76.2%. A total of 105 sites, 18.9%, were readable by neither. Seventy-six, or 13.7%, challenged or refused both profiles. These are one-day home-page observations, not proof that an entire store was inaccessible.[1]
The 330-merchant assessment found an evidence problem
A second sample covered 330 mostly retail merchants selling in the United States and Canada. Two lists came from a commercial partner operating a consumer shopping agent, and a third contained large North American retailers. Each merchant contributed one scanner-selected offer page per profile, plus linked home and policy pages.
The mandate asked for one item delivered to ZIP code 10001 within ten days, within a budget of the displayed price plus 25%, and returnable within 14 days. Across two profiles, 38 checks each created 25,080 statuses. Deterministic rules—not a language model—assigned pass, fail or unknown.
Median evidence coverage was about 82%. Only 103 HTTP and 102 headless profiles cleared the benchmark's 90% overall and 80% per-pillar thresholds; 115 merchants qualified under at least one profile and 90 under both. The thresholds themselves are design choices, not validated predictors of successful purchases.[1]
All-in cost was the hardest material term
Material terms had the lowest coverage. Under the HTTP profile, the all-in total was determinate at 18 merchants and failed at 178. Tax drove 137 failures because it was unstated, lacked a rate or was deferred to checkout; shipping drove another 41.
Among profiles with enough coverage to rank, the median score was high—89.0 for HTTP and 90.2 for headless—but the material-terms pillar averaged only about 62 to 64 while four other pillars averaged at least 95. That contrast shows why a single readiness number can be misleading.
For a shopper, the distinction is practical. A human may continue after seeing a base price and learn the final amount later. An agent told never to exceed a budget should stop when it cannot bound the total, or ask the person for approval. Guessing turns missing disclosure into financial risk.[1]
Why the findings cannot support a merchant ranking
The four full assessments used one mandate each, one observation and one rater. The same AI assistant helped write the rubric, collected user-present evidence, adjudicated verdicts and audited the census. No inter-rater agreement was measured, and the paper acknowledges that at least one refund judgement was contestable.
Rules were developed from those cases and have not been tested by a second rater on unseen examples. Weights are unfitted, no sensitivity analysis was performed and the projected clearing rule is uncalibrated. Nothing links a failed check with an actual abandoned purchase, refund, dispute or financial loss.
The 555-site census came from one location on one day and used unsigned, unregistered automated profiles. It cannot distinguish an agent-specific block from a general automation, geographic or network restriction. Offer and checkout pages can behave differently from a home page, and a verified agent identity might receive different access.[1]
Commercial interests and responsible use
All authors are affiliated with Evolutionairy AI, which develops RAILS and has a commercial interest in merchant assessment. A commercial partner supplied two merchant lists. The authors say no merchant, processor or partner funded, directed or reviewed the study, but the interest makes independent validation especially important.
The team released a de-identified dataset with rubric, aggregate results and capture hashes, while the scanner is available on request. Merchant identities were withheld because findings rested on single observations. That is appropriate restraint, but it prevents outsiders from checking individual classifications against the live pages.
The paper correctly says a readiness record should inform rather than replace a person's approval and must not release a payment. Merchants can still act on the mundane problems it identifies: state tax and shipping assumptions earlier, keep policy pages consistent and offer verifiable, time-stamped terms.[1]
What would change the assessment
A stronger study would repeat observations across regions and dates, compare declared, signed and ordinary user traffic, and sample offer through checkout without placing orders. Independent raters should apply a frozen rubric to unseen merchants and publish agreement and sensitivity analyses.
The benchmark then needs outcome validation. With merchant or processor cooperation, researchers could test whether access, decision sufficiency and evidence classes predict task completion, human intervention, cancellations, refunds, disputes and unauthorised purchases while preserving consumer approval.
Until then, the evidence supports a practical but limited conclusion: many sites are technically readable to declared agents, yet missing totals and policy detail can prevent a cautious agent from proving that a purchase meets instructions. It does not establish which merchants are ready or which should be trusted.[1]
What this means for people
- A shopping agent should stop or ask when tax, shipping or refund terms cannot be bounded, rather than infer that a displayed base price is the total.
- Merchants can improve both human and agent decisions by publishing consistent all-in costs and return conditions before checkout.
- No automated readiness score should replace buyer approval or be treated as proof that a transaction will succeed without dispute.
Global context
The evidence is limited to merchants publicly associated with two processors and mostly North American retailers observed from Ontario. Access controls, taxes, disclosure rules and shopping practices differ internationally, so the figures should not be generalised to global commerce.
What the evidence does not yet show
- The preprint is company-authored, commercially interested and not peer reviewed.
- Samples were non-random, mostly North American and observed once from one network location.
- The rubric, weights, thresholds and clearing bands are unvalidated design choices.
- The detailed pilot lacked independent raters and used an AI assistant in design, adjudication and audit.
- No checkout, completed purchase, consumer harm, dispute or financial outcome was measured.
What to watch next
- Independent application of a frozen rubric to unseen merchants
- Repeated access measurements with signed agent identities and geographic controls
- Outcome data linking evidence gaps to human intervention, refunds and disputes
- Merchant standards for verifiable totals, policy terms and buyer approval
Living evidence record
Impact record IAI-0650YOG
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Not yet
Record status
Monitoring
Last checked
14 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 14 October 2026
Research question, topics and regions
Whether merchant websites let declared shopping agents read offers, determine material terms and retain evidence of what a buyer agreed
Topics: AI agents · Online shopping · Consumer protection · Price transparency · Web access · Payments
Regions: United States · Canada · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
The Impact Brief · Free
Keep following the stories that matter to you.
Get our weekday briefing on what changed and why it matters. Choose your topics below.
Continue the story
There’s more to this story
Finance & Business
Should an AI agent be allowed to move a company's money?
Airwallex says its revamped business accounts let approved agents manage liquidity and transfers within rules and approvals. The 30 September release identifies important controls, but provides no independent test, loss data or deployment denominator.
6 min · 2 sources
Finance & Business
What would SAP gain from TechWolf?
SAP has signed an agreement to acquire TechWolf, whose software maps tasks, skills and labour-market signals for workforce planning. The price is undisclosed, closing remains conditional and no public evidence yet shows that deeper integration will improve hiring, reskilling or job design.
7 min · 2 sources
Finance & Business
How much do investment advisers disclose about using AI?
A peer-reviewed study found affirmative AI-use language in 23.7% of 388 US adviser brochures and risk language in 27.6%. It measures formal disclosure—not underlying adoption, safe performance or AI-washing.
8 min · 2 sources
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.