A learner has worked through every mock exam in your course, and the real one is still three weeks away. What they need now is a new set of questions on the section they keep failing.
Your writers already have a backlog, and each new item needs an expert, an editor and a round of fixes before it’s ready.
With AI in assessment, a first draft takes minutes.
Quality is the open issue. A practice question that’s too easy or simply wrong teaches the wrong lesson, and learners pay for it on exam day. Anyone selling practice tests has a stake in where AI drafts hold up and where they slip.
The past two years of research point to an answer. It’s a yes with strings attached, and the strings decide how a test prep company should let AI into its question writing.
What Is AI in Assessment, and Why Does It Matter in 2026?
Think of AI assessment in education as machine learning applied to tests: models that write questions, deliver them, score answers and make sense of the results. One quiz item counts. So does the feedback page a learner opens after a three-hour mock exam.
In test prep, the list gets shorter. Practice banks, timed mocks, instant scoring, a line explaining every wrong answer. All of them run on a steady supply of well-written items. That supply is where AI test prep promises the most.
Learners already study with AI on their own. Inside Higher Ed asked 1,047 US college students about it in July 2025. Most of them, 85%, had leaned on generative AI for some class assignment that year. Close to half, 46%, put exam and quiz prep among their main reasons.
College students aren’t certification candidates yet, though plenty will be in a few years, and they’ll bring the habit with them. Once someone is used to getting fresh AI questions for free, a fixed bank in a paid course starts to look thin.
Keeping up by hand doesn’t scale, least of all for the summative assessment, the graded final mock exam that has to match the real one.
How Does AI Generate Exam Questions?
Most AI-drafted exam questions come out of one of three setups.
Experts can build templates with variables, a language model can write from source text, or a pipeline can mix both and add human review. What separates them is control: how much a person fixes in advance, and how much the model writes on its own.
Templates don’t need large language models (LLMs), the systems behind ChatGPT, while the other two do.
Template-Based Automatic Item Generation
This one predates language models entirely. An expert writes item models, which are questions with fixed wording and a few blanks. Take a dosage problem. The blanks hold a patient’s weight, a drug concentration and an ordered dose, and software plugs in valid numbers and works out the right answer for each version.
A single model can yield hundreds of versions of one question. Licensing-exam publishers have relied on this approach, also called automated item generation, for years, because every output inherits the expert’s logic. Because the stem (the question’s main text) barely changes, psychometricians can predict each version’s difficulty. Those are the people who measure test quality.
The limit is range: template-based generation only produces variations on questions experts already designed.
Good for dosage math or accounting. Weak for anything that hinges on a long reading passage or a judgment call.
LLM-Based Question Generation
A language model can write a complete multiple-choice question from a passage of course material. It drafts the main text, the correct answer, three or four wrong options and a short explanation.
The wrong options, called distractors, are the hardest part: each has to look plausible to a learner who half-knows the topic, and none can be arguably correct. The explanation becomes the answer rationale that appears once an answer is submitted.
The weak point is where the model’s facts come from. Left alone, an LLM draws on its general training data, which may be outdated or may not match how a specific exam board phrases a rule.
That risk shrinks with retrieval-augmented generation (RAG). Before writing, the system looks up passages in the provider’s approved material, say this year’s study guide, and the model sticks to them. Fewer invented facts. Difficulty and tone still need someone’s eye. Each draft can then cite the page it came from, which speeds up expert review.
Hybrid Pipelines: Item Models + LLMs + Human Review
Most production setups combine both methods and put people at fixed checkpoints. Experts define the structure of a question type, and an LLM varies the scenario wording, which template software handles poorly.
A second model pass can take on distractor generation, proposing wrong options based on mistakes the provider has seen learners make. Some pipelines also let one model critique and revise another’s draft before a person sees it, which catches clumsy wording cheaply.
Every draft then goes to a reviewer before it enters the item bank, the stored pool of approved questions. Neither software nor a model can tell whether a draft matches what the exam actually tests, and that’s the call a reviewer is there to make.
How Good Are AI-Written Questions? What the Research Shows
Good enough, with conditions. Well-built AI-generated questions can sort strong learners from weak ones as well as standardized-exam items, but they tend to run easier than the exams they imitate, and some drafts carry errors only a subject-matter expert will catch.
A large field study and Pearson VUE’s research show both sides.
Where They Hold Up
The largest real-world test so far ran in 91 college classes across the US. Researchers from Harvard, Stanford and Microsoft Research studied 1,686 students, and 1,208 of them, in 71 classes, took exams written by an LLM pipeline. They analyzed each question with item response theory (IRT), a statistical method that estimates how hard a question is and how sharply it sorts students.
Discrimination, the second measure, shows how well an item tells apart students who know the material from those who don’t. On that measure, 36% of the AI-written questions fell into the study’s two highest bands. Only 21% of the standardized-exam items did.
For a test prep provider, discrimination is the property that counts. A question that splits prepared learners from unprepared ones tells both the learner and the course team something true about readiness.
Pearson VUE reached a similar result for driving theory tests. In its 2023 studies, drafts from popular AI tools matched unedited human-written items on factual accuracy, correctly marked answers and the quality of the wrong options.
Pearson VUE expects LLM drafting to make drafting much faster, though it hadn’t measured the saving yet.
Where They Fall Short: Weak Distractors, Ambiguity, Factual Errors, Bias
Difficulty is the most consistent miss.
In the Harvard, Stanford and Microsoft Research study, students answered about 60% of the AI-written questions correctly, against 39% for the standardized ones, so the AI-written sets ran easier. For test prep, that gap costs something: a learner who scores well on easy practice walks into the actual test overconfident.
Pearson VUE’s studies point in that direction too. Requested and actual cognitive level, the type of thinking a question demands, didn’t consistently match, for AI and human items alike. Prompts asking for more difficulty produced longer drafts that weren’t harder, and large batches held near-duplicates.
Other failures show up item by item:
- weak distractors a learner can rule out without knowing the topic
- ambiguous wording that makes two options defensible
- factual errors and AI hallucinations, such as an invented regulation or a wrong threshold
- names, settings or cultural references that put some learners at a disadvantage
Each failure needs its own check, run by someone other than the model that wrote the item.
How Psychometric Validation Works
Whoever wrote a question, pilot data shows how it performs. The standard route is pilot testing: new items appear unscored inside live practice exams, so real learners answer them before they count.
Two numbers come out of a pilot. The p-value is the share of learners who answered correctly. A second figure, the discrimination index, compares how often test-takers with high and low overall scores got the item right; a low or negative value means the question trips up people who know the material.
Numbers alone don’t replace expert review of the content. Together, the two build the case for validity: evidence that a question measures the skill it claims to measure.
There’s APA guidance on AI in testing too, from the American Psychological Association’s committee on tests and assessment. Eight areas are on its list. Two of them, human oversight plus bias and fairness, map straight onto AI-drafted items.
For an exam prep course, oversight comes down to a name and a record: someone signs off on each question, and its performance data stays on file.
Where Does AI Add the Most Value for Exam Prep Providers?
Once quality is under control, the obvious next question is where the payoff is.
For exam prep providers, AI in assessment adds value in four places, and each one changes something for both the learner and the course team.
Scaling Practice Question Banks Without Scaling the Writing Team
Fresh items are the raw material of a prep course. A certification candidate who has finished a course’s practice tests still has weak sections to drill, and someone preparing for a retake needs a full set they haven’t seen. A repeated question loses its value once the answer is memorized. With AI drafting, writers set the topics and a difficulty target, then review each draft and fix or reject it.
Review becomes the bottleneck, so it needs planning, because unchecked drafts can’t go live. Once it’s planned, the item bank grows faster while every item still passes an expert. Learners get fewer repeats and deeper coverage of their weak topics.
Adaptive Practice and Personalized Feedback
Every answer a learner gives shows something about what they know, and adaptive systems use those answers to pick the next question.
Three ethics misses in a row bring easier ethics practice, while mastered topics come up less often. AI keeps the pool of reviewed items large enough for each weak spot to get practice the learner hasn’t seen.
Explanations target the specific wrong option picked, which makes this formative assessment: practice that guides study with no grade attached. These answers can also feed predictive analytics for education that flag learners at risk before exam day.
The limit is supply, since adaptivity needs reviewed items at more than one difficulty level for every topic.
Automated Scoring and Performance Analytics Across Attempts
A single score tells a learner little about what to change before the next try. Comparing attempts does more. Tracking error patterns across attempts shows whether someone keeps missing one topic or runs out of time in the last section, two problems with different fixes.
High Pass Education helps US mental health professionals get certified. AnyforSoft built the assessment and reporting layer of its custom exam prep platform, where learners see trends across attempts and administrators see which answer learners pick most on each question and how long they spend on it.
Instructors use these numbers to adjust course material and test content.
Such reports set the bar for AI assessment features, which earn their place when their score reports name the topic behind each lost point.
AI Tutors Between Practice Attempts
Learners get stuck on one concept and may wait days for an instructor’s next session. A tutoring assistant built on the course material can step in between two practice tests: it explains the concept another way and links the lesson that covers it. Instructors then lose fewer hours to routine clarifications.
The design limit is what makes it safe. The assistant answers only from approved course content and hands it off to an instructor when a question goes beyond it, while a general chatbot can make up rules. Anything it writes during a session stays practice-only until an expert reviews it.
How Do You Check Item Quality Before Learners See It? A 6-Step Workflow
Calibration and expert review, the conditions the research sets, are what turn AI-generated test questions into items a provider can stand behind.
The work runs in six steps, each handing a concrete output to the next.
Step 1 — Define the Test Blueprint and Item Specifications
Before a model drafts anything, the team writes down what the exam actually measures. The blueprint lists the content domains and how much of the official test each one takes up, so practice sets keep that balance.
A spec sheet for each item type then fixes the format and the reading level. It also sets the target thinking skill, often expressed through Bloom’s taxonomy levels such as recall or application.
This is the step where construct alignment gets decided: each question tests a skill named in the blueprint and nothing outside it. The output is the blueprint document plus a spec sheet per item type, so later steps don’t start from scratch. They reuse both as prompt input and as review criteria.
Step 2 — Generate Items From Your Own Approved Content
Drafting starts from the provider’s approved study material. The team picks source passages for each blueprint domain and writes prompts that carry the spec sheet’s rules, plus examples of strong and weak items.
That prompt engineering work, writing and testing the model’s instructions, is reusable. Once a prompt reliably yields usable drafts, it joins a shared library.
Owned or licensed material also keeps the provider clear of disputes over the copyright of source material, which matters when questions are sold as part of a paid course. What comes out is a versioned library of templates and tested prompts, so any item can be traced back to the prompt and passages that produced it.
Step 3 — Run Automated Checks First
Step 3 ends in an automated report, built before any expert sees a draft. Scripts catch the cheap problems, so an expert doesn’t spend time on them. A batch goes through:
- a near-duplicate scan against items already in the bank
- a check that the answer key marks exactly one correct option, sometimes by having a second model answer the item without seeing the key
- a reading-level check against the spec sheet
- flags for names, places or contexts that may carry bias
These checks map to the failures Pearson VUE and the field study found: duplicates from large batches, ambiguous keys, wordy items and bias. Weak wrong options and factual errors slip past most scripts, so they’re left for the expert in Step 4.
A draft that fails goes back to Step 2, and repeated failures point to a prompt in the library that needs rewriting. The report shows reviewers which drafts failed and why.
Step 4 — Route Every Item Through Subject-Matter Expert Review
Learners see nothing until a content specialist signs off. People call this human-in-the-loop review: the model drafts, and a qualified person keeps, edits or bins each item.
The rubric starts with content accuracy. Then come the key, which has to be unambiguous, the wrong options, and how well the item fits the blueprint.
A second reviewer often handles bias separately. In that fairness and bias review, they look for wording or scenarios a learner from another background could read another way. The step leaves behind the rubric and a log of who approved what and which changes they made.
Step 5 — Pilot Items and Run Item Analysis
Expert approval still needs backing from learners. In field testing, the pilot described under psychometric validation, new drafts run unscored inside regular practice exams until each reaches the response count set in the spec sheet.
A psychometrician or the team’s data analyst then runs the numbers. The output is the two figures explained earlier, item difficulty (p-value) and the discrimination index, for every piloted question, with a decision for each: keep, revise or retire. Pilot results also feed back into Step 1, since a domain where drafts keep failing usually points to a spec sheet that needs tightening.
An item that’s too easy or too hard goes back to expert review, and one that strong test-takers keep missing usually leaves the bank.
Step 6 — Monitor, Version and Retire Items in Production
Once a question goes live, its numbers drift. An item whose answer circulates among learners gets easier, which is an academic integrity problem as much as a quality one. Recalculating the pilot figures from live answers on a set schedule exposes that drift.
Each edit creates a new version, so results point to the exact wording a learner saw. Retirement rules pull a question from rotation after a set number of exposures, or when its difficulty shifts past a threshold or the content outline changes. With items saved in QTI, the common exchange format for test content, the item bank can move platforms later. The workflow ends there, with a versioned bank and retirement rules on paper.
Should You Build or Buy AI Assessment Tools?
Running the six-step workflow takes tooling.
Ready-made products are quicker and cheaper while volumes are small. A custom layer costs more at the start.
What it buys is ownership of the items and of the learner data. Bank size settles a lot of it.
So does how exam-like practice must be, and which systems, like the LMS and CRM, need the results.
When an Off-the-Shelf AI Exam Question Generator Is Enough
Small courses rarely need more than a ready-made product. The usual AI tools for test prep are quiz features built into learning platforms, separate authoring apps, and chat assistants that draft on request. Each is basically an AI question generator. They handle end-of-lesson checks and low-stakes quizzes well, plus rough first drafts an instructor cleans up.
The fit is strongest when the team has no engineers and the bank is small enough for one person to review. It also helps when practice doesn’t need to copy a specific exam’s timing and scoring.
The limits appear as the course grows. These tools tend to store items in their own format, so moving a bank out means manual work. Few keep the pilot statistics, version history or approval log the six-step workflow relies on, and learner data stays in the vendor’s system. For a provider whose product is exam readiness, those gaps tend to show once the bank outgrows a single reviewer.
When a Custom Assessment Layer Pays Off
Building starts to pay off when practice has to feel like the official exam. That means test modes with matching timing and navigation, scoring that follows published score ranges, difficulty that adapts between sections, and reports that compare a learner’s results across attempts.
Ownership is the second reason. The provider keeps its item bank as its own intellectual property, along with the approval history behind each item.
Integration is the third. Results flow into the LMS. Enrollment data comes in from the CRM, the system that tracks prospects and students, so instructors and sales staff share records. Learner data also stays inside the provider’s own compliance setup, which matters under data privacy rules such as FERPA and GDPR.
A custom layer doesn’t have to replace the existing LMS. High Pass Education kept its LMS, and AnyforSoft added the assessment and reporting layer on top, so course delivery stayed as it was while testing got deeper.
Assessment Platforms in Practice: AnyforSoft Case Studies
Two AnyforSoft projects show how a custom assessment platform serves exam prep, both on the custom route: one added to an existing LMS, one replacing paper testing.
High Pass Education
Mental health professionals need a certification exam before they can open a private practice, and High Pass Education prepares them for it. Its LMS already ran realistic tests. To make preparation more personal, the company wanted test rules set for each learner and reports that follow progress from one attempt to the next, so instructors could guide people between tries.
AnyforSoft built an assessment and reporting layer on top of the LMS the company already had. It adds:
- per-learner time limits, attempt counts, grading rules and passing thresholds
- speedrun tests that train speed and accuracy under time pressure
- comparative reports that show each learner’s results across attempts
- interfaces aligned with WCAG 2.1 accessibility standards
Together, they turned the LMS’s existing tests into rehearsals the team can tune for each learner.
The client reported its strongest Q1 to date in 2025, and the business can now take on more students without hiring more instructors.
For a test prep company planning to grow enrollment, the parallel is staffing: per-learner test rules and cross-attempt reports take on work instructors would otherwise do by hand.
Digital Testing Platform for a US Tutoring Company
Since the 1980s, a US company has prepared students for standardized tests and college admission. Its SAT prep ran on paper, with printed booklets, manual grading and reports written by hand.
AnyforSoft built a digital environment modeled on Bluebook, the official SAT practice app.
It’s built on an Open edX core extended with custom Python and React code, and includes:
- three test modes, from fully timed to self-paced
- module-to-module adaptive difficulty based on results in the previous module
- a grading scheme that reports a score range, similar to the SAT
- an automatic PDF report per test, with the time spent on each question
- a proctor cabinet for in-person proctoring sessions
- data exchange with Salesforce
Grading and reporting now run mostly without manual work. The Digital Assessment Platform shows that a test-specific environment can spare a tutoring company hours of manual work while students practice under conditions close to the real SAT.
Why EdTech and Exam Prep Teams Choose AnyforSoft
AnyforSoft is an EdTech software development partner with a track record in testing and reporting systems for exam prep companies. Five qualities shape how the team approaches assessment projects.
15+ Years Building Education Platforms
Since its founding in 2011, AnyforSoft has focused on education software development. The team has delivered more than 150 education-focused projects and built over 10 AI products for education.
Assessment features get built by engineers who have shipped learning products before and know how learners use them.
Assessment Engines That Work With Your Existing LMS
Plenty of test prep companies already have a working learning platform and no reason to leave it. AnyforSoft builds testing layers that connect to that system.
For High Pass Education, a change to a user’s record reaches the connected LTI tools (outside tools linked through a shared standard) in 5 to 20 seconds.
The team also handles wider LMS integration work across open-source and commercial platforms.
AI Grounded in Your Own Content
Wittenborg University of Applied Sciences uses an AI assistant built by AnyforSoft to answer applicants’ and students’ questions.
It answers only from the university’s official pages and documents, so when staff edit a page, the assistant’s answers follow.
Question writing benefits from that grounding, because drafts built from approved material are easier to check.
In an exam workflow, responsible AI starts there, as it does in the AI solutions for education AnyforSoft builds, where each answer traces back to approved sources.
Integrations That Keep Data in Sync
Test results are useful only when they reach the people who act on them. For the SAT testing platform, AnyforSoft built CRM integration with Salesforce that works in both directions. The system pulls student and enrollment data from the CRM, processes test results and sends them back under rules the client defined.
Flexible Engagement Models
Work can start at whatever stage a provider has reached. Teams still shaping an idea begin with discovery, which covers product analysis, UX design and an MVP. Teams with a roadmap move straight to building and launching, and live platforms get ongoing support. Companies with their own engineers can add a dedicated development team from AnyforSoft.
Frequently Asked Questions
What is AI in assessment?
It’s the use of machine learning to write, deliver, score or analyze exams and quizzes. In test prep, that mostly means drafting practice questions and grading attempts, with experts reviewing every draft.
Can AI write exam-quality test questions?
Yes, once a qualified person reviews them. In a 2025 field study, questions drafted by AI separated strong and weak students at least as well as standardized-exam questions, though they ran easier. The quickest proof for a provider is to mix AI-drafted items unscored into a mock session and compare their statistics with the bank’s human-written ones.
What is automatic item generation?
It’s a method where experts write question templates with variable parts and software fills the variables to create hundreds of valid versions. Each version gets a correct answer, calculated automatically. The upfront cost is the expert’s time to build each template, so the method pays off only for question types a bank needs in large numbers.
How accurate are AI-generated questions compared with human-written items?
They can come close, though in the 2025 field study AI items ran easier than the standardized ones, so difficulty needs checking. Factual accuracy is a separate matter: a model can state an outdated rule with full confidence, so checking facts against the provider’s own material stays a human job. Whether an item measures consistently, its reliability, shows up only in pilot statistics.
What checks should an AI-drafted exam item pass before it goes live?
Five, each tied to a step of the workflow:
- a match with the exam blueprint and the item spec
- automated checks for duplicates, a single correct option, reading level and bias
- logged approval from a subject-matter expert
- a pilot with real learners
- acceptable difficulty and discrimination statistics
An item that doesn’t pass one of them goes back for revision or leaves the bank.
Does AI test prep work for high-stakes exams like the SAT or licensing exams?
It works when practice copies the test closely. SAT practice has to copy the timing, the module structure and score reporting. With licensing and certification exams, the official content outline is the guide, and every item gets checked against the rules in force today. AI makes drafting faster. The stakes are why expert review and pilot data can’t be skipped.
Do subject-matter experts still need to review every question?
For anything scored or exam-like, yes. Review does get quicker: experts start from a draft, and scripts have already thrown out the obvious failures. Self-check quizzes are different. Some providers review only a sample there and let its error rate decide how much to trust the rest.
Can AI question generation and scoring plug into an existing LMS?
Yes. The drafting service usually runs on its own and talks to the LMS over standard connections. Both Moodle and Canvas accept external tools as plug-ins. Activity data can move as xAPI statements, which record what a learner did and when. The harder part is the data model: scores, attempts and item versions need one consistent record across systems.
Which AI models and tools work best as an AI exam question generator in 2026?
There isn’t one winner. General-purpose LLMs from OpenAI, Anthropic and Google can all draft multiple-choice items. Hand them source material and a detailed spec, and the drafts get much better. Assessment platforms build drafting into their authoring tools, as Learnosity does with Author Aide and Questionmark does with its AI authoring features. [Editorial note: Learnosity, “Learnosity launches Author Aide…” (learnosity.com/edtech-blog); Verdict, “Questionmark launches AI tool to enhance enterprise workplace assessments”; verify product names before publishing] For low-stakes practice, a simple AI quiz generator is enough. For exam prep, the deciding factors are grounding, review tools and whether items can be exported.
How do large language models write distractors and answer explanations for multiple-choice questions?
It works backward from the right answer. Say the concept is a dosage formula: likely mistakes include flipping the ratio or applying the adult rule to a child, and each mistake becomes a wrong option. Good prompts ask for distractors that are believable, plainly wrong to an expert and about as long as the key, since one long option gives itself away. Then comes answer explanation generation, where the model writes a short note for every option. A learner who picked B reads why B fails. Experts still check both, because a ‘wrong’ option sometimes turns out to be arguable.
