What an AI voice agent can and can't do for your business
An AI voice agent can answer every call, handle routine questions, take messages and book appointments in a natural-sounding voice, day or night. It still struggles with strong accents, noisy lines, complicated requests and upset callers, so it needs a clear way to hand over to a person. Used for the routine calls, it works well.
That's the honest summary. The technology has improved quickly, and the best voice agents now hold a perfectly normal conversation. But a phone call is a harder job than a chat window, and a business that knows where the limits are will get much more out of one.
How does an AI voice agent work?
Your phone number is connected to the agent through a telephony provider, so calls ring through to it like they would to a person. From there, there are two common designs:
- A chain of three steps. Speech-to-text turns what the caller says into text. A language model, given your business information and rules, decides what to say. Text-to-speech reads the reply aloud in the voice you've chosen. Telephony platforms such as Twilio offer this as a service, handling the speech side while your own application decides the replies.
- One speech-to-speech model. A single model listens to the audio, decides what to do and speaks the reply. OpenAI's documentation describes this as one model that interprets audio, decides and responds in speech, while the chained approach gives more control over each stage, such as keeping transcripts or checking a reply before it's spoken.
Either way, the agent can be connected to your calendar, your CRM or your job system, so it can check a slot, book it and send you a summary when the call ends.
Sources: Twilio ConversationRelay documentation; OpenAI voice agents guide.
What is an AI voice agent genuinely good at?
- Answering every call. It never has a queue, never goes on lunch and doesn't mind a 9pm call on a Sunday.
- Routine questions. Opening hours, areas covered, whether you do a particular job, rough prices you've given it, how to find you.
- Taking a proper message. Name, number, address, what the job is and how urgent it is, written up and sent to you by text or email straight away.
- Booking. Offering real slots from your calendar and confirming one, for a salon appointment, a quote visit or a call back.
- Overflow. Picking up when you're on another call or with a customer, rather than letting it ring out.
- Consistency. It gives the same accurate answer on the hundredth call as on the first.
A plumber under a sink can't answer the phone. A voice agent that takes the details of a leaking boiler, works out that it's urgent and texts the plumber straight away is genuinely useful.
Where do AI voice agents still trip up?
This is the part sales pages tend to skip.
- Accents and dialects. Speech recognition is much better than it was, but strong regional accents, fast speakers and unusual words still cause mistakes. A misheard word can change the meaning of a whole request.
- Background noise. A caller in a van, on a building site, in a busy shop or on a bad mobile signal is harder to understand. So are two people talking at once.
- Names, addresses and numbers. Postcodes, house numbers, email addresses and surnames are where errors creep in. Letters that sound alike are easy to confuse on a phone line.
- Complicated requests. "I need the boiler looked at, but also the radiator upstairs, and can you quote for moving the bathroom, but not until after Christmas" is a lot to unpick. A person handles that better.
- Emotion and urgency. An angry customer, a distressed caller or a genuine emergency needs a human. The agent should recognise this and hand over, not keep asking questions.
- The rhythm of conversation. Knowing when someone has finished speaking, coping with interruptions and avoiding awkward pauses has improved a lot, but it isn't perfect.
- Callers who won't talk to a machine. Some people will hang up as soon as they realise. That's their choice, and it's one reason the agent should offer a person early.
Why does a voice agent need a human handover?
Because some calls will always need you, and a voice agent with no way out does more harm than no agent at all. A good handover has three parts:
- Clear triggers. The caller asks for a person, the agent isn't confident it understood, the request is outside what it's allowed to handle, or the caller sounds upset or the matter is urgent.
- A real route. Transfer the call live if someone's available. If not, take the details and promise a call back with an honest time, not "as soon as possible".
- A summary. Whoever picks it up gets the caller's details and what was said, so the customer never has to repeat themselves.
How do you try one safely?
Start small and widen it as you gain confidence.
- Begin with overflow and out-of-hours calls only. You keep answering when you can; the agent picks up the calls you would have missed anyway.
- Say it's an AI at the start of the call. It's honest, and it lets callers ask for a person straight away if they'd rather.
- Read back the important details. Have the agent repeat the phone number and spell back the postcode before it ends the call.
- Limit what it can do. No discounts, no promises about arrival times it can't check, no taking card payments over the phone.
- Review every call for the first few weeks. Listen to the recordings or read the transcripts, and fix anything it got wrong.
- Treat recordings as customer data. Call recordings and transcripts contain personal data, so they need the same care under UK GDPR as the rest of your customer records, including telling callers they're recorded.
A useful first test: before it goes near a real customer, ring it yourself from a noisy place, mumble a postcode, change your mind halfway through and ask for a person. Then ask a friend with a different accent to do the same. You'll learn more in ten minutes than from any demo.
Is a voice agent right for your business?
| Likely a good fit | Probably not yet |
|---|---|
| You miss calls while working with your hands or with customers | You rarely miss a call |
| Most calls are questions, bookings or messages | Most calls are long, technical conversations |
| Out-of-hours calls are common | Your customers almost always email or message |
| Each missed job is worth a lot | Callers are often vulnerable or distressed |
If most of your enquiries arrive by website form or email rather than phone, start there instead. A text-based AI receptionist is simpler, cheaper to run and easier to check. We explain the differences in what is an AI receptionist, and compare AI with human options in AI receptionist vs answering service vs virtual assistant.
If you want to see whether a voice agent would cope with your calls, our AI agents and automation service starts with a small proof on your real calls before any full build.
Quick answers
What can an AI voice agent do?
An AI voice agent can answer every call, handle routine questions, take detailed messages and book appointments into your calendar, day or night. It works best for overflow and out-of-hours calls.
What are the limitations of AI voice agents?
They still struggle with strong accents, background noise, names and postcodes, complicated multi-part requests and upset callers. That's why every voice agent needs a clear way to hand the call to a person.
How does an AI voice agent work?
Your phone line connects to the agent through a telephony provider. Most designs either turn speech into text, have a language model decide the reply and read it aloud, or use a single speech-to-speech model.
Should a voice agent say it's an AI?
Yes. Saying so at the start is honest and lets callers ask for a person straight away, and if calls are recorded, callers need to be told.
