Navigation Menu

Click the "+" to see inside a chapter or use the search to the right.


Versions Compared


  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Modify Statistics and Comments for Outcomes


  1. Select the Manage → Setting → Review Workflow menu.



    Allow users to review their own Questions

    Select Yes to allow authors to review Questions they have written.

    Number of reviewers needed to advance a Question

    The number of Reviewers who must advance a Question in its current form before it is approved. Allowed values range from 0 to 15; the default is 3.

    Setting this to 0 causes the review process to be entirely skipped.

    Number of reviewers needed to return a Question

    The number of Reviewers who must return a Question in its current form before it is returned to the author. Allowed values range from 1 to 15; the default is 3.

    Require reviewers to classify unclassified Questions to level

    Choose Yes if Reviewers should ensure the Question is correctly classified, and then select the level to which the Question should be classified from the drop-down list.
    Lowest Level in the drop-down list indicates that in a Blueprint of varying depth, the lowest level must be selected irrespective of the depth.


    When Yes is selected for Require reviewers to classify unclassified Questions to level setting on the Review Workflow page and a Question with one of the Review States having Unclassified Blueprint or Blueprint level lower than the one specified in the drop-down setting on that page is opened on the Examine Questions page, then the following error message is displayed in red text. The State of such Questions cannot be changed to anything other than one of the available Review States or to the Obsolete state.

    Require Comment on modifying Question

    Select Yes if the Reviewer should record/explain changes made on the Create/Edit Question page.

    Allow Reviewers to Edit

    Allow or disallow users with Reviewer permission to edit the following areas: Item Text (Stem and Option text), Blueprint, Metadata, and Reference.

    Each of these areas has check boxes next to them, which grant edit permissions to Reviewers if selected and deny them if not selected. All the check boxes are enabled by default, thus allowing Reviewers to edit all these areas unless manually deselected by the Manager.

    A Manager is also bound by the permissions of a Reviewer on the pages affected by the above settings.



    A Question cannot be modified by a Reviewer if all the check boxes are not selected and an error message is displayed in red text.



    A Reviewer is able to partially modify a Question if check boxes against some of the above items are not selected, thus allowing modifications to only certain areas of the Question. A message in red text informs the Reviewer about areas that cannot be edited.

    If Metadata and Blueprint are set to not editable, then a disallowed icon  (red circle icon with a diagonal slash across it) is displayed just after the configuration name. Moving the mouse cursor over these icons shows the message Manager has disabled editing this content within the current review state.


    An error message in red is displayed if you try to disallow Reviewers to edit Metadata in Review Workflow when it is set to Mandatory under Manage → Settings → Metadata. If there are multiple Review States, then at least one Review State must allow the Reviewers to edit Metadata.


    When the check box next to Blueprint is removed in Review Workflow, meaning it cannot be edited by Reviewers, and you try to enforce a Blueprint level against Require Reviewers to classify unclassified Questions to level, then you get an error message in red.


    When you clear the check box for Reference in Review Workflow so that Reviewers should not be able to edit Reference in Questions, you will get an error in red text if it has been set as mandatory (check box next to Questions must have references is enabled) in the  Questions tab of Manage → Settings → Configuration.

    Show 'Retain Statistics' pop-up warning for modified items 

    The popup to confirm retaining statistics while trying to save a Question with statistics after modifying for a Review State will not be shown, if this check box is not selected, and the statistics will be retained.

    This check box is selected by default, meaning the pop-up to confirm retaining statistics appears while modifying Questions with statistics.

    This check box is grayed out and the statistics are retained while saving a Question with statistics after modifying for a Review State, if the Show 'Retain Statistics' pop-up warning for modified items check box at the Project level is not selected.

    OutcomesThe Outcomes list what happens when you advance or return a question. There can be multiple advance and return states. As of the 1811 release, you are not able to specify if Comments are required, or specify whether Statistics are required. In a future release the Statistics and Comments may be modified.
    Only Managers can access this page to modify these settings.
  2. After selecting all information, click Update to save changes or Cancel to discard changes.


  1. Click the Add new outcome button for the desired review state.
  2. Click the Choose state drop-down list and select the desired outcome from the list.

    The new Outcome is added to the list.
  3. All Return Outcomes require reviewers to add a comment explaining why they are returning the question. You can specify if Comments are required for any of the other Outcome options. Click the Comments drop-down list for the newly added Outcome and select Required.
    Image Added
  4. For each of the Outcome states, you can specify whether to Always retain statistics or let the reviewer decide by selecting Always retain.
    Image Added
  5. Click Update. The Outcomes are updated for the review state. You are not able to change the existing outcome, but you may delete it. 

Delete Review Outcomes

  1. Click the Delete  icon for the outcome you wish to remove from the review state. The outcome is removed from the review state.
  2. Click Update to save the changes.


  1. Select Select review states. The page with the list of Available Review states and the Current Order for that Review State is displayed.
  2. Deselect the  check box beside the Review State you would like to disable for the current Project and click Save. The Review State is disabled for the current Project and is no longer visible on the Review Workflow page.
  3. Then click Update to configure the changes.
    If the Review State you want to delete has any assignments, it shows an error message: Please delete the assignments for the states to delete.
    Image Modified