ServiceNow Automated Test Framework and Release Evidence

The Automated Test Framework (ATF) is ServiceNow’s built-in test tool: you assemble tests from steps such as “Impersonate”, “Record Insert” and “Open a New Form”, group them into suites, and run them on a sub-production instance. Building a test that goes green is easy. Building one that would go red when the behaviour breaks, and whose result an auditor or CAB can rely on, takes more thought. That second kind is what interviewers mean when they ask how you use ATF for release confidence.

How ATF runs a test

A test is an ordered list of test steps. Each step is a step configuration (the reusable definition, for example “Record Query”) plus runtime data (which table, which conditions). Steps fall into two families:

• Server steps run directly on the instance: impersonating a user, inserting, updating or querying records, running a server-side script, making REST calls. They don’t need a browser.
• Client steps drive a form in the classic UI: open a form, set field values, check whether a field is visible or mandatory, submit. These need a client test runner, which is simply a browser tab logged into the instance and waiting for work.

A test suite groups tests and can be scheduled. After each test the framework rolls back the data the test created, so the instance should look the same before and after. Results, including step logs and screenshots for client steps, are stored as test and suite result records and are retained for 30 days by default.

Test execution is controlled by system properties (sn_atf.runner.enabled and sn_atf.schedule.enabled), and the documented intent is that tests run on non-production instances. Running ATF on production means running impersonation, inserts and rollbacks against live data, and most teams keep it switched off there.

Picking behaviour worth testing

The most valuable ATF tests describe a business rule in a sentence and fail when that sentence stops being true. “A purchase request over 1,000 USD needs the requester’s manager to approve before fulfilment starts” is a good test subject. “The Short description field exists” isn’t.

A good test exercises the path a real user takes, with the roles a real user has, and checks an observable outcome. For the purchase request rule, a server-only test could look like this:

1. Create a User step: a requester with only snc_internal, and a manager referenced from the requester’s manager field.
2. Impersonate the requester.
3. Record Insert on u_purchase_request with amount 1,800, with Enforce security ticked so the insert obeys the requester’s ACLs.
4. Record Query on sysapproval_approver: one row, approver is the manager, state is Requested.
5. Impersonate the manager, then Record Update the approval to Approved.
6. Record Validation on the purchase request: approval is Approved and state has moved to Fulfilment.

Then add the cases that should fail. Insert a request for 600 USD and assert that no approval row exists. Impersonate an unrelated user and assert that the approval update is rejected. Negative cases catch the regressions that matter most: approvals that silently disappear and access that silently widens.

The Enforce security option deserves emphasis. Without it, record steps behave like server code, which doesn’t consult ACLs, so a step can insert or update a record the impersonated user could never touch through a form or the API. A test with security off answers “does the data model allow this?”, not “can this user do this?”.

Scripted assertions

When the built-in steps can’t express a check, the Run Server Side Script step can. The default template wraps your code in a function that receives outputs, steps, params, stepResult and assertEqual. An illustrative check that the approval reason captured the amount the manager reviewed (written for this article, not run on an instance):

(function(outputs, steps, params, stepResult, assertEqual) {
    // sys_id of the Record Insert step from step 3
    var requestId = steps('9f1c2a7d1b2e3010a5b8c3d4e5f60718').record_id;

    var appr = new GlideRecord('sysapproval_approver');
    appr.addQuery('sysapproval', requestId);
    appr.query();
    assertEqual({ name: 'one approval row', shouldbe: 1, value: appr.getRowCount() });

    appr.next();
    var reason = appr.getValue('approval_reason') || '';
    assertEqual({ name: 'reason quotes the amount', shouldbe: true, value: reason.indexOf('1,800') > -1 });
})(outputs, steps, params, stepResult, assertEqual);

Two things in that script are worth defending in an interview. It reads the request’s sys_id from the earlier step’s output rather than querying for “the latest purchase request”, which would break the moment two tests run in parallel. And it asserts on the meaning (the reason mentions the amount) rather than an exact string, so harmless wording changes don’t break the build. The field name approval_reason is an assumption for illustration; check the actual column on your instance.

What rollback doesn’t undo

Rollback covers ordinary records the test inserts, updates or deletes, but ServiceNow documents a list of tables that are excluded, along with any table that extends them. The list includes sys_email, ecc_queue, syslog*, sys_history_line and sys_number_counter. A short Node.js script using that documented list shows what a test run leaves behind:

const EXCLUDED = ['sys_history_line', 'ecc_queue', 'sys_email', 'sys_email_log', 'report_executions',
  'report_stats', 'sys_execution_tracker', 'sys_progress_worker', 'sys_schema_change',
  'sys_upgrade_history', 'sys_upgrade_history_log', 'sys_mutex', 'sys_plugin_log', 'sys_status',
  'sys_number_counter', 'sys_amb_channel_presence'];
const PREFIXES = ['syslog', 'sys_amb_message', 'sys_cluster', 'cmdb_metric', 'ts_', 'v_', 'sys_delete_recovery'];

const survives = (t) => EXCLUDED.includes(t) || PREFIXES.some((p) => t.startsWith(p));

// Records written while the purchase-approval test ran
const touched = { u_purchase_request: 1, sysapproval_approver: 3, sys_email: 2,
  ecc_queue: 1, syslog: 14, sys_user: 2, sys_number_counter: 1 };

for (const [table, n] of Object.entries(touched))
  console.log(`${table.padEnd(22)} ${String(n).padStart(2)}  ${survives(table) ? 'NOT rolled back' : 'rolled back'}`);

u_purchase_request      1  rolled back
sysapproval_approver    3  rolled back
sys_email               2  NOT rolled back
ecc_queue               1  NOT rolled back
syslog                 14  NOT rolled back
sys_user                2  rolled back
sys_number_counter      1  NOT rolled back

The consequences are concrete. The two notification emails exist after the test, and if outbound email is enabled on that instance, they were sent. The ECC queue record means an outbound integration call through a MID Server may already have happened, and rollback won’t recall it. The number counter isn’t restored, so the next real request skips a number, which confuses anyone expecting gapless numbering. Point outbound REST messages at a mock endpoint on test instances, keep email delivery off there, and assert on the queued email record rather than on someone’s inbox.

Timing and flaky tests

Not everything happens inside the step that triggers it. Async business rules run later from the scheduler, and record-triggered flows run in the background by default. A Record Query placed straight after the insert can run before the flow has created the approval, and the test fails one run in five. Adding a fixed wait hides the problem until the instance is busy.

Better options: test the synchronous part (the before or after rule) in one test, and the flow’s logic in isolation with known inputs in another; or make the assertion tolerate eventual completion with a bounded retry in a scripted step, and fail with a clear message when the bound is reached. Client steps add their own flakiness (form layout, the runner tab going to sleep), so keep them for what only a form can show, such as UI policy behaviour, and do the rest server-side.

Turning results into release evidence

A green run on a developer’s instance isn’t evidence. Evidence is a suite run on the target test instance, after the update sets or application version were committed, by a scheduled or pipeline-triggered job, with the suite result linked to the change request. Because results are kept for 30 days by default, export the suite result or attach it to the change before it ages out; a CAB asking about a release from last quarter shouldn’t be told the proof was cleaned up.

Before an upgrade, run the same suites on a fresh clone of production before and after upgrading it. A test that passed before and fails after points at either an upgrade change or a customisation the upgrade skipped, and both are things you want to know before production does.

Interview practice

Your ATF suite for a catalog item passes every night, yet production users report that the item’s approval sometimes never appears. The test impersonates an admin, inserts the requested item with a Record Insert step, and immediately queries for an approval. How would you fix the test so it would have caught the problem?

Answer

The test has three blind spots. Impersonating an admin bypasses the ACLs and admin-only data that a real requester lacks, so I’d create or pick a requester with realistic roles and turn on Enforce security in the record steps. Inserting the record directly skips the path real users take (ordering through the catalog, which triggers the catalog flow), so I’d order through the catalog steps or trigger the same flow. And querying immediately races the background flow; when the approval appears it passes, and when the flow is slow the real defect and a timing artefact look the same. I’d separate the synchronous checks from the flow checks and use a bounded wait with an explicit failure message.

Then I’d add the negative cases (a request below the threshold has no approval; an unrelated user can’t approve) and a case reproducing the production report, for example a requester with no manager, which is a common reason approvals never generate. The fix is accepted when that new test fails on the current build and passes after the change.

Related reading

• ServiceNow interview questions
• ServiceNow multiple-choice questions
• ServiceNow Update Sets and Conflicts, on getting the code under test onto the test instance
• ServiceNow Scheduled Jobs, for the background work that makes tests timing-sensitive
• Tables excluded from rollback (ServiceNow documentation)