Responsible AI Auditor practice test

Sample questions

Two questions from this practice test, in the same style as the certification exam. Answer before opening the explanation.

1. Question 1

   An auditor must conclude whether the content filter of a customer-facing chatbot blocks requests for self-harm instructions, as the organization's AI policy requires. Which source of evidence is the most reliable on its own?

   1. The QA team's latest test report, signed by the head of QA, showing that all of its test prompts were blocked
   2. A walkthrough in which the responsible engineer explains the filter and shows the policy settings screen
   3. Test prompts that the auditor submits to the production system in a controlled session, observing the replies
   4. A written representation from the product owner confirming that the filter has been active since launch
   Show answer and explanation

   Correct answer: C. Test prompts that the auditor submits to the production system in a controlled session, observing the replies

   Evidence the auditor obtains directly, by reperforming the control on the system actually in use, is generally more reliable than evidence produced or asserted by the auditee. The QA report is useful, but it was produced by the auditee, and its prompts and test environment would first have to be evaluated; it may not reflect production. A walkthrough shows how the filter is designed and configured, not that it blocks real requests. A written representation is an assertion by management and cannot replace evidence. The auditor's own test shows operation at one point in time, so for a whole period it is combined with other evidence, such as monitoring records.

2. Question 2

   A junior auditor drafting the working papers for an AI management system audit labels as "findings" only the points where requirements were not met. Under ISO 19011, how should the team leader correct this understanding?

   1. Findings result from evaluating evidence against criteria and can show conformity or nonconformity, and lead to improvement opportunities
   2. Findings are the team's overall statement on whether the audit objectives were met, given after considering all the work
   3. Findings are the verifiable records, statements of fact and other information collected that relate to the audit criteria
   4. Findings record only nonconformities, while points of conformity are kept as supporting notes outside the audit report
   Show answer and explanation

   Correct answer: A. Findings result from evaluating evidence against criteria and can show conformity or nonconformity, and lead to improvement opportunities

   In ISO 19011, audit findings are the results of evaluating the collected audit evidence against the audit criteria. They can indicate conformity or nonconformity, and they can lead to the identification of opportunities for improvement or the recording of good practices. The overall statement on the audit objectives is the audit conclusion, reached after considering all the findings. Verifiable records and statements of fact relevant to the criteria are the audit evidence, the input to findings. Restricting findings to nonconformities is exactly the junior auditor's misunderstanding: conformity is also a finding.