A good answer should evaluate a defined task against a human-reviewed reference set and measure correction effort. Faster drafting is not the acceptance criterion by itself.
Why it matters to buyers. NIST’s Generative AI Profile identifies confabulation: output can be inaccurate or internally inconsistent while appearing credible. That is directly relevant to evaluating generated summaries and mappings, but the NIST publication is not a certification of any GRC product. (nvlpubs.nist.gov)
Put it to: Apostol Vassilev, Norman Marks and the buyer’s AI-risk lead.
A strong answer should propose a bounded test: 1. Select a task, such as drafting a control mapping or summarising an evidence package. 2. Establish an expert-reviewed reference and acceptable outcomes, including when the system should decline to conclude. 3. Include missing evidence, conflicting documents, obsolete policy versions and misleadingly similar controls. 4. Check whether each material claim points to the correct source and version. 5. Measure unsupported conclusions, missed contradictions, reviewer corrections and total task time. 6. Record the product, model/configuration information available, test date and retest triggers.
Ask for proof: a result register with case-level examples—not an unsupported overall accuracy percentage. Separate drafting, retrieval, mapping and approval tasks rather than averaging them into one score.
Follow-up: “Which error would make you disable the feature even if the average time saving looked attractive?”
Expert quote — pending interview: [Insert approved quotation and attribution.]
Sources cited on this page
- nvlpubs.nist.gov ↗nvlpubs.nist.gov