Use one buyer-owned demonstration script across the shortlist. Follow supplier demonstrations with hands-on testing of the highest-risk requirements in the proposed configuration.
Record the product release, installed modules, user roles, configuration and licence assumptions behind every result. ServiceNow’s current GRC documentation, for example, distinguishes business-user, approver and other roles, with some roles dependent on installed applications and entitlements. A successful administrator demonstration is therefore not sufficient evidence of an occasional contributor’s experience. (servicenow.com)
Which scenarios should every finalist demonstrate?
| Scenario | Proposed acceptance evidence |
|---|---|
| An overdue action | The correct owner receives escalation; extensions require the agreed approval; the decision remains traceable |
| A disputed risk score | The original assessment, challenge, approver and final decision remain distinguishable |
| Restricted evidence | An unauthorised user cannot retrieve the item through normal screens, search, reports, exports or AI features |
| A failed integration | The failure is visible, assigned and recoverable without silent loss or duplicate records |
| Evidence reuse | The same evidence can support permitted uses while preserving scope, period and access restrictions |
| An administrator change | Your administrator changes an owner, field or approval step and demonstrates testing and rollback |
| A management report | A reported figure can be traced to the relevant records and evidence |
| Bulk export | Records, files and relevant history can be retrieved and interpreted outside the platform |
How should the evaluation be scored?
Establish mandatory gates first: security, required access separation, regulatory applicability, critical workflows and any essential deployment restrictions. A high aggregate score must not compensate for failing a mandatory gate.
A suggested weighting for the remaining comparison is:
- Workflow fit: 30%.
- Migration and integration feasibility: 25%.
- Contributor usability and administration: 20%.
- Three-year commercial fit: 15%.
- Portability and exit: 10%.
These are starting weights, not research-derived optimums. Adjust them before seeing supplier results.
Label evidence consistently: buyer-tested, supplier-demonstrated, documented, roadmap or not demonstrated. A missing demonstration is not proof of a missing capability, but it remains an unresolved purchase risk.
What additional checks apply to AI features?
NIST’s Generative AI Profile identifies risks including confidently incorrect output, sensitive-data disclosure and inappropriate human reliance. Use those risks to design buyer-specific acceptance tests rather than accepting a general claim that a feature is responsible or secure. (nvlpubs.nist.gov)
- [ ] Require source references for generated mappings and summaries.
- [ ] Test incomplete, contradictory and stale evidence.
- [ ] Repeat retrieval tests with differently authorised users, including after access is revoked.
- [ ] Require human approval before material changes to controls, findings or risk decisions.
- [ ] Establish prompt, output and document retention; model-provider access; training use; and deletion terms.
- [ ] Measure review and correction time as well as generation time.
Which US sector questions need specialist review?
| Buyer | Question to resolve before commitment |
|---|---|
| Banks | Apply the institution’s third-party risk process to the platform and delivery providers. The interagency community-bank guide covers planning through termination and states that outsourcing does not remove the bank’s responsibility; it is supervisory guidance, not a new law or compliance safe harbour. (occ.gov) |
| Healthcare covered entities and business associates | Determine whether the proposed service will create, receive, maintain or transmit ePHI on the organisation’s behalf. HHS says such cloud arrangements require an appropriate business associate agreement, including where the provider stores encrypted ePHI without the key. (hhs.gov) |
| Internal-audit-led programmes | Have the chief audit executive review independence, access, training and technology governance. The IIA’s standards address organisational independence and require collaboration with IT and information security when implementing technology. (theiia.org) |
These examples are not a complete applicability assessment. Do not automatically apply banking guidance to insurers or assume every healthcare GRC deployment contains ePHI.
Output: an evidence-backed scorecard, unresolved-issues register and shortlist decision.
Proceed when: mandatory scenarios pass in the intended operating context, or an authorised decision explicitly addresses a permissible exception.
Hold when: essential functionality depends on an unspecified future release, unpriced custom work or permissions that ordinary users will not have.
Sources cited on this page
- servicenow.com ↗www.servicenow.com
- nvlpubs.nist.gov ↗nvlpubs.nist.gov
- occ.gov ↗www.occ.gov
- hhs.gov ↗www.hhs.gov
- theiia.org ↗www.theiia.org