What is AI Chatbot Reply Optimization (ACRO)?
Goodbye SEO, Hello ACRO
Have you heard of the term AI Chatbot Reply Optimization (ACRO)?
Probably not because I just invented it. ACRO refers to the process of improving the visibility of your website/company in AI chat bot replies.
Just like how the SEO industry was created to game search engine rankings, I predict that the ACRO industry will be birthed to influence the replies that LLMs, such as ChatGPT, Perplexity, and Gemini, provide to their users.
How does AI chatbot powered by a Large Language Model (LLM) work?
In simple terms, an AI chatbot powered by a Large Language Model (LLM) works like this:
Learning from data: The LLM is trained on a huge amount of text data from books, articles, websites, and more. During training, it learns patterns, grammar, facts, and even some reasoning by predicting the next word in a sentence based on the previous words.
Understanding Prompts: When you ask the chatbot a question or give it a command (this is called a "prompt"), the LLM processes the text and tries to understand what you're asking or saying.
Generating Responses: Based on its training, the LLM generates a response by predicting a sequence of words that make sense as a reply to your prompt. It doesn't actually think or understand in the way humans do but uses the patterns it learned during training to produce human-like text.
Providing Output: The generated response is then shown to you as the chatbot's reply.
Disclaimer that this is an oversimplified explanation, I’m not an engineer, let alone one well versed in LLMs.
Experimenting with AI chatbot replies - What is the best accounting software in Australia?
As AI Chatbot Reply Optimization (ACRO) is still a nascent industry, I decided to conduct my own fieldwork. Specifically, I asked four LLMs (ChatGPT, Gemini, Perplexity, and Meta AI), the question — “What is the best accounting software in Australia?” and here’s what they replied.
ChatGPT
ChatGPT listed the following options in order:
Xero
MYOB
QuickBooks
Reckon
Saasu
Zoho Books
Frustratingly, unlike the three other LLMs I tested, ChatGPT didn’t provide sources. And when prompted to list it’s sources, it didn’t, instead provide a generic reply — see image below.
I decided to take it a step further and just directly ask ChatGPT to pick the best accounting software for me. It replied with a request for more details, which I provided. And Xero was determined the best option.
It would seem that Xero has the best ACRO for ChatGPT.
Winner: Xero
Perplexity
Perplexity listed the following options and provided two sources for each.
MYOB
Xero
QuickBooks
FreshBooks
Zoho Books
ReckonOne
Saasu
The best frequently cited source was - 7 Best Accounting Software For Small Business In Australia (2024)
It appears MYOB landed at the number 1 spot thanks to the source mentioned above and the Canstar article titled Best-Rated Accounting Software
Interestingly, second placed Xero had two sources that were completely different from MYOB.
Just like for ChatGPT, I asked Perplexity to pick the best accounting software for me and it replied that MYOB as the Best Overall Accounting Software with the following sources to back up it’s response:
6 Best Accounting Software for Australian Businesses in 2024
Best accounting software in Australia for small businesses in 2024
After a closer examination of the sources, the article - 6 Best Accounting Software for Australian Businesses in 2024, which Perplexity cites twice to justify picking MYOB does not mention MYOB in the article at all. It would appear there’s some hallucination going on.
Perplexity also listed Xero, QuickBooks, and FreshBooks as alternatives. See image below for the response.
Despite the hallucination, it would appear MYOB has the best ACRO for Perplexity.
Winner: MYOB
Gemini
Gemini listed Xero, MYOB, and QuickBooks as the top contenders. It also followed up with Reckon One, Sage, and FreshBooks, as notable options.
Similar to Perplexity, Gemini also lists it sources but it doesn’t attribute the specific sources to each reply — see image below. And unlike the other LLMs, Gemini includes the websites of the accounting software as sources.
When asked to pick the best accounting software for me, Gemini asked for several follow up questions — see image below. It also provided a quick overview listing Xero, MYOB, and QuickBooks.
Once I provided my hypothetical answer — I run a digital marketing agency with 5 people and budget of $100 per month — it listed the following options in order:
Wave accounting
Zoho Books
FreshBooks
Just as I was about to close Gemini and move onto the next LLM, I decided to be thorough and ask it to list it sources for picking Wave Accounting.
What I received was a correction and an apology from Gemini, as Wave Accounting was no longer available in Australia. Oops, it would seem like Google is still struggling with the innovator’s dilemma that I outlined in my article — Is AI a product or a feature?
It also listed three alternative options: Rounded, Thriday, and Xero.
Based on Xero appearing in the top position twice and also in the last reply, I will have to award Xero with the best ACRO for Gemini.
Winner: Xero
Meta AI
Meta AI listed the following options — see image below.
MYOB
QuickBooks
Xero
FreshBooks
Zoho Books
In regards to sources, Meta AI provided only two:
And when I asked it to pick the best accounting software for me, it replied with Xero and six reasons to back up it’s choice.
Like the previous times, I asked Meta AI to also list it sources for Xero, which it replied with 6 sources but for some reason there was an error with the links — see image below.
Winner: Tie between MYOB and Xero
Based on this very rudimentary research I conducted, it would appear that Xero and MYOB are neck and neck when it comes to AI Chatbot Reply Optimization (ACRO), with QuickBooks in third place.
Although it wasn’t the intention of this research but it does appear that ChatGPT and Perplexity are significantly ahead of Gemini and Meta, at least when it comes to researching the best accounting software in Australia.
How would AI Chatbot Reply Optimization (ACRO) work?
Disclaimer that the tactics below are speculative and not something I’ve actually tested.
The most obvious way that ACRO could work is by targeting the training data. Unfortunately, most AI companies don’t disclose the exact data that their LLMs have ingested. But there are some open source datasets such as Common Crawl that we know LLMs including GPT-3, LLaMA have utilised. Using the accounting software example, it seems that winning the Canstar Blue award for Best-Rated Accounting Software is a decisive factor for LLMs in determining their ranking.
Another way to ensure your company/website is included in the training data is through PR and branded content. It’s been reported that OpenAI has signed licencing deals with publishers such as The Financial Times. Again, using the accounting software example, we can see that the Canberra Times article — Best accounting software in Australia for small businesses in 2024 contributes to rankings and is actually branded content paid for by MYOB.
In a similar vein, OpenAI’s deal with Reddit means the social network is another avenue for ACRO practitioners to game. Perhaps AstroTurfing on Reddit will be back in vogue.
Another tactic that could be deployed is generating responses favourable to your company/website. Imagine hiring hundreds of contractors who reply to an AI chat bot that “Actually, QuickBooks is the best accounting software in Australia.” It’s conceivable that this could influence the LLM’s ranking in the future and move QuickBooks to 1st or 2nd place.
Challenges for AI Chatbot Reply Optimization (ACRO)
The lack of transparency of the training data is the main challenge for ACRO practitioners. But over time, I do believe that we will gradually uncover how LLMs determine their replies.
Unlike SEO, where there is only one dominant player (Google), ACRO would need to either factor in several LLMs, at least for now.
I hope you enjoyed reading this article on AI Chatbot Reply Optimization (ACRO), if you did it would mean a lot to me if you shared this with a friend!
Things that caught my attention
How Costco Hacked the American Shopping Psyche - The New York Times
Lenny’s podcast with Deb Liu (CEO of Ancestry, ex-Facebook, PayPal, eBay)
I have seen the term "share of model" used to describe what % of models cite your brand - definitely something people are thinking about!