13 min read
TL;DR Tap to expand the short version
- In March 2024 the SEC brought its first AI washing enforcement actions against two investment advisers, who paid 400,000 dollars in combined civil penalties for claims about AI stock research they could not substantiate.
- Backtested returns are classified as hypothetical performance under the SEC Marketing Rule, and the SEC’s position is that hypothetical performance is generally not appropriate in advertising aimed at a general audience.
- Peer reviewed mathematics explains why, strong simulated performance is easy to produce by testing a modest number of strategy variants, and almost nobody reports how many variants they tried.
- Four questions separate a reading tool from a return claim. All four can be answered from a product page in about five minutes.
Somewhere in the last two years, AI stock research stopped being unusual. People paste a 10-K into a model and ask what the risk factors say. They ask it to compare three companies in the same sector on the same five metrics. That is a reasonable way to use AI to research stocks, and it is not what this article is warning about.
Running alongside it is a second category of product, sold with the same three letters, that implies something completely different. Regulators have been unusually blunt about that second category, and they have already taken money off firms for it.
The useful skill is telling them apart on sight. It is easier than it sounds, because the distinguishing feature is not how sophisticated the model is. It is what kind of claim is being made.
This article covers how to read a claim. It is not investment advice, and it does not evaluate any named product as a buy or a sell.
Two Types of AI Stock Research
Sort every AI investing product you have seen into one of two piles.
Comprehension tools help you process information that already exists. Summarising a filing, extracting a number from a footnote, translating accounting language, comparing disclosures across companies. The claim being made is about reading speed. Nobody is promising you an outcome.
Performance claims assert or imply a result. A percentage return, a hit rate, a chart of what the strategy would have done. The moment a number like that appears in marketing, a body of securities regulation switches on, and the rule applies whether the underlying technology is a neural network or a spreadsheet.
That regulatory line is the whole point. You do not need to assess the model. You need to notice which pile the marketing has put itself in.
Where AI Research Helps
The distinction is useful because the comprehension pile is genuinely useful and the same regulators who warn about AI fraud also point retail investors toward the primary documents these tools read.
Every US public company files its annual and quarterly reports into EDGAR, the SEC’s full text search system. It is free, it is complete, and almost nobody outside the profession uses it, mostly because a 10-K runs to a hundred pages of dense language designed by lawyers to be precise rather than readable.
That is a reading problem, and reading problems are what these models are good at. Summarising the risk factors section. Pulling the segment revenue table out of a filing. Explaining what a particular accounting treatment means in plain terms. Comparing how three companies describe the same competitive threat.
There are two limits. A language model does not verify anything, so if it misreads a number, nothing in the process catches it, which is why the source document is the thing to check against rather than the summary. And summarising a filing tells you what a company said about itself. It is not independent analysis, and the filing is written by people with an interest in how it reads.
Used that way, the tool is doing something checkable. The moment the output shifts from what the document says to what the stock will do, you have crossed into the other pile.
Why Backtests Can Mislead
A backtest applies a strategy to historical data to show what it would have returned. It sounds like evidence. Mathematically, it is closer to a search result.
The clearest treatment of why sits in a 2014 paper in the Mathematical Society paper by David Bailey, Jonathan Borwein, Marcos López de Prado and Qiji Jim Zhu, under the title Pseudo-Mathematics and Financial Charlatanism. It is not a polemic. It is a proof.
Three findings help explain performance charts
- High simulated performance is easy to reach. The authors demonstrate that you only need to test a relatively small number of alternative strategy configurations before one of them produces an excellent backtest, purely by chance.
- More testing makes it worse, not better. The higher the number of configurations tried, the greater the probability that the winning backtest is overfit to noise in the historical data rather than capturing anything real.
- The number that would let you judge is almost never published. The paper notes that analysts and academics rarely report how many configurations they tried, which leaves an investor with no way to assess how much overfitting is baked into the result they are being shown.
The uncomfortable part is the fourth finding. The authors show that where markets have memory effects, an overfit strategy does not simply revert to zero performance out of sample. It produces negative expected returns. Their suggestion is that this helps explain why so many systematic funds fail to deliver what their backtests promised.
Put plainly, a beautiful backtest is not weak evidence of a good strategy. Without knowing how many strategies were tested to find it, it is not evidence at all.
Modern AI makes this worse in one specific way. Testing thousands of configurations used to take real time and real compute. It no longer does. The search space got cheaper, which means the probability that any given published backtest is the survivor of a very large search went up.
What Regulators Have Enforced
This is not a theoretical concern that regulators might one day get to. There is a paper trail.
The AI washing cases. On 18 March 2024 the SEC announced settled charges against two investment advisers, Delphia (USA) Inc. and Global Predictions Inc., for false and misleading statements about their use of AI. Delphia had told clients its collective data made its AI smarter, able to predict which companies and trends are about to make it big
. Global Predictions described itself as the first regulated AI financial adviser and advertised expert AI driven forecasts. Neither firm could substantiate the claims. Delphia paid 225,000 dollars, Global Predictions paid 175,000 dollars, and both were censured, without admitting or denying the findings. These were the SEC’s first enforcement actions of their kind against investment advisers.
The backtesting rule. Under the Marketing Rule, Rule 206(4)-1 of the Investment Advisers Act as amended in December 2020, backtested returns fall inside the definition of hypothetical performance, alongside model portfolio results and projected returns. Specifically, performance produced by applying a strategy to data from periods when the strategy was not in use.
Advisers are not banned from showing it. They are required to have written policies ensuring the hypothetical performance is relevant to the likely financial situation and investment objectives of the intended audience, and to give that audience enough information to understand the criteria, the assumptions, and the risks and limitations involved.
The commercially significant part is the SEC’s position on who the intended audience can be. As Marketing-rule analysis, the SEC has taken the view that hypothetical performance is not appropriate for advertisements directed at a mass audience or intended for general circulation, on the reasoning that unsophisticated investors generally do not understand its limitations.
Read that against a public product page showing backtested returns to anyone with a browser, and you can see why this became an enforcement theme. In a sweep announced on 11 September 2023, the SEC charged nine registered investment advisers over hypothetical performance advertised to the general public without the required policies in place, with penalties ranging from 50,000 to 175,000 dollars, Goodwin analysis. The charges weren’t about the numbers being wrong. They were about showing those numbers to the wrong audience without the required safeguards.
The joint alert. The SEC’s Office of Investor Education and Advocacy, FINRA and the North American Securities Administrators Association issued a joint Investor fraud alert. Its first substantive point is not about technology at all. It is that securities laws generally require firms, professionals and platforms to be registered, and that a promoter’s lack of registration should trigger further investigation before any money moves.
Who Needs to Act
The four-question framework below works for anyone, but the stakes and the fix are different depending on which side of the product you sit on.
| If you are… | What is at risk | The one thing to do this week |
|---|---|---|
| A retail investor evaluating a tool | Mistaking a backtest for a track record, and sizing a position on a number nobody has proven repeats. | Run the tool’s marketing page through the four questions before funding an account, not after. |
| A marketer at a registered adviser | Personal, firm-level liability. The SEC’s March 2024 and September 2023 actions were brought against named advisers, not against the AI vendors they used. | Pull your site’s performance pages and check every one against the Marketing Rule’s audience test, before compliance does it for you. |
| A product lead at a fintech building AI research tools | Building a feature that quietly becomes a performance claim the moment marketing screenshots it, without anyone in engineering intending that. | Add a data-boundary disclosure (source, date range, backtested vs. realised) as a required field on any output that includes a number, not an afterthought bolted on later. |
Four Checks for Performance Claims
None of this requires understanding the model. Four questions, answerable from a product page and two free lookups.
1. Backtested or Realised?
Why? these are different categories of evidence. Realised performance is what happened to actual money. Backtested performance is what a simulation says would have happened.
How to check look for the words backtested, simulated, hypothetical or model, usually in smaller text below the chart. If the page does not say which it is, treat it as backtested, because a firm with realised returns has every reason to say so.
2. Count the Variants
Why? this is the single number that determines whether a backtest means anything, and per the Notices paper it is almost never disclosed.
How to check ask. A firm that has thought seriously about overfitting will have an answer and may reference deflated Sharpe ratios or out of sample testing. A firm that has not will treat the question as strange. Both responses are informative.
3. Check Registration and Marketing
Why? registration is the baseline the joint Investor Alert leads with. And a registered adviser publishing backtested returns openly to anyone is doing the exact thing the SEC has repeatedly brought actions over.
How to check search the firm on Adviser disclosure database or via Investor.gov. Both are free and take under a minute.
4. Identify Missing Information
Why? every model has a data boundary. A tool built on filings does not see next week. A tool trained to a cutoff date does not know what happened after it. Products that state their boundaries are describing a reading tool. Products that imply no boundary are making a prediction claim.
How to check look for a stated data source and date range. Its absence is the finding.
The pattern underneath all four is the same one that shows up wherever AI meets a regulated outcome. The question is never whether the technology is impressive. It is whether the specific claim in front of you can be evidenced, and by whom, and to what standard.
Take the last AI investing product you saw advertised and run the four questions against its landing page. If it answers all four, you are looking at something built by people who expect to be asked. If it answers none, you have learned something useful for the price of five minutes.
FAQ
No, provided you are using it to read rather than to predict. Summarising filings, extracting figures and comparing disclosures are checkable tasks, and the source documents are free on the SEC’s EDGAR system. The caution applies to products that assert or imply a return.
AI washing is the SEC’s term for making false or misleading claims about the use of artificial intelligence. On 18 March 2024 the SEC settled charges against two investment advisers, Delphia and Global Predictions, which paid 225,000 and 175,000 dollars in civil penalties respectively and were censured, without admitting or denying the findings.
Because strong simulated performance is easy to produce by testing a modest number of strategy variants, and the number tried is almost never reported. Research published in the Notices of the American Mathematical Society in 2014 proved this formally, and found that under memory effects an overfit strategy can produce negative expected returns out of sample.
Yes, under conditions. The SEC Marketing Rule treats backtested returns as hypothetical performance and requires written policies ensuring relevance to the intended audience, plus disclosure of criteria, assumptions, risks and limitations. The SEC’s position is that hypothetical performance is generally not appropriate in advertising aimed at a mass or general audience.
Search the firm on Investment Adviser Public Disclosure at adviserinfo.sec.gov or through Investor.gov. Both are free. The joint Investor Alert from the SEC, FINRA and NASAA treats a promoter’s lack of registration as a prompt for further investigation before investing.
Spotted an error? Contact the desk. Our corrections policy explains how we handle corrections and updates.