AI Assistants at Work: What They Can and Cannot Do Reliably


AI assistants are reliable coworkers for some jobs and unreliable ones for others. The dividing line is simple: they are strong at drafting, summarizing, and organizing, and weak at verified facts, citations, and numbers. Use them where mistakes are cheap and easy to spot. Keep a human in charge where mistakes are expensive and hard to notice.

That rule matters more now because assistants are no longer an experiment. Stanford's 2025 AI Index reports that 78% of organizations used AI in at least one business function in 2024, up from 55% in 2023, and generative AI use in at least one function more than doubled, from 33% to 71%. A Federal Reserve Bank of St. Louis analysis of worker self-reports found that people using generative AI saved an average of about 2.2 hours per week. The question is no longer whether to use an assistant. It is which tasks to hand over.

What AI Assistants Reliably Do Well

Assistants are at their best when the output is a starting point, not a final answer. In these jobs, a human review is fast because errors are visible on the surface.

  • First drafts. Emails, memos, job descriptions, and status updates. The assistant supplies structure and wording; you supply judgment and facts.
  • Summaries. Long documents, meeting transcripts, and research threads condensed into key points. You can check a summary against the original in minutes.
  • Brainstorming and reformatting. Variations on a headline, a table turned into bullets, a rough outline expanded into sections.
  • Routine code and search. Boilerplate code, explanations of error messages, and searches that return sources you can open and read yourself.
  • Everyday coordination. Drafting replies, preparing agendas, and organizing notes. The cost of a bad draft here is a rewrite, not a wrong decision.

The pattern is consistent: assistants save the most time on text-heavy work where you already know enough to judge the result. That is also why the St. Louis Fed analysis found time savings concentrated among people who actually used the tools during their work hours.

Where They Fail, With Numbers

The failures are not random glitches. They come from how these systems work: they predict the most likely sequence of words, not the verified truth. When a prompt demands a specific answer, they often produce a plausible one rather than admitting they do not know.

A March 2025 study by Columbia University's Tow Center for Digital Journalism, published in the Columbia Journalism Review, tested eight AI search tools on citation accuracy. Researchers gave each tool excerpts from 200 news articles and asked it to identify the original headline, publisher, date, and URL. Across 1,600 queries, the tools gave incorrect answers more than 60% of the time. Perplexity did best and still failed 37% of the time; ChatGPT Search failed 67% of the time, and Grok-3 failed 94% of the time. A link next to an answer, in other words, is not proof the answer is grounded.

Reasoning models show the same problem in a different form. OpenAI's own system card for its o3 and o4-mini models reported hallucination rates of 33% and 48% on the PersonQA factual benchmark, compared with 16% for the earlier o1 model. A longer, more confident, more "reasoned" answer is not automatically more reliable. In specialized domains the error rates climb further; legal questions are among the worst, because a fabricated case citation looks exactly like a real one.

None of this makes assistants useless. It makes them the wrong tool for one specific job: producing facts you cannot check.

Which Tasks to Hand Over, and Which to Keep

Use this split when you decide what to delegate:

Hand it over: first drafts, summaries of documents you have, brainstorming, reformatting, and explanations of concepts you already understand.

Check before you use it: any number, date, quotation, citation, or claim about a specific person, company, or event. Open the source yourself. If the assistant cannot point to a source you can open, treat the claim as unverified.

Keep it human: anything that leaves the building as fact — a report to a client, a filing, a medical or legal statement, a price or contract term, and any decision where a wrong answer costs money, reputation, or safety.

A Four-Step Check That Takes Five Minutes

For any assistant output you plan to rely on, run these four checks before you send it on:

  1. Open the source. Click through to the page or document. If the link is dead or the page does not say what the assistant claims, discard the claim.
  2. Check the numbers separately. Recompute or re-verify every figure from the original source, not from the assistant's summary of it.
  3. Verify quotes word for word. A quote that sounds perfect is the most likely to be invented. Search for the exact phrase.
  4. Ask a second source. For anything important, confirm it with a source that does not come from the assistant — an official page, a primary document, or a second tool.

Teams that use assistants this way get the time savings without the failure mode: the assistant does the fast, visible work, and the human owns the facts. That is the whole skill. Assistants do not need to be perfect to be useful. They need a workflow that assumes they are sometimes wrong.

Sources

Comments

Popular posts from this blog

AI and Jobs: Which Tasks Change First, According to Researchers

What Marketing Tasks Will AI Replace Next — and Which Still Need a Human?

How AI Screens Your Resume Before a Human Sees It — and How to Write for It