Most integration bugs aren’t about getting the request right. They’re about what happens when the network does something rude: the remote system takes 40 seconds, the connection resets, the response never arrives. A ServiceNow incident that should open one ticket at a vendor opens three, or a change that should have been pushed to a deployment tool silently isn’t. Interviewers asking about integration failures want to hear that you design for the uncertain middle case, not just success and failure.
Three outcomes, not two
When an outbound call returns 201 Created, the remote work happened. When it returns 400 Bad Request, it didn’t. But when the call times out, you don’t know. The remote system may have received the request, created the ticket and then lost the connection while replying. Or the request never arrived. From ServiceNow’s side, both look the same: no HTTP status, an error message such as a read timeout, and haveError() returning true on the response object.
The tempting fix is “just retry”. For a read, that’s fine. For “create a ticket”, retrying an operation that already succeeded creates a duplicate. So the design question is how to make a repeat of the same logical operation harmless, which is what idempotent means.
Where the call should run
Before retries, placement. A call from a before or after business rule runs inside the user’s save: the user waits for the remote system, and if it hangs, the save hangs with it. Worse, the remote side effect can’t be rolled back if the save later fails. Outbound work belongs outside the transaction:
- an async business rule or a flow running in the background for simple cases;
- for anything that needs reliable retries, an outbox: the save writes a row to a small queue table (operation, target record, payload, attempt count, next attempt time, state), and a scheduled job or flow processes pending rows.
The outbox makes every attempt visible and restartable, and it survives the remote system being down for an hour. Set an explicit timeout on every call (setHttpTimeout on RESTMessageV2) so a hung endpoint doesn’t hold a worker indefinitely. Calls through a MID Server go via the ECC queue and complete asynchronously, which adds its own latency to plan for.
Retrying without duplicates
There are two standard ways to make “create ticket” safe to repeat. If the remote API supports it, send an idempotency key: a value that identifies the logical operation (not the attempt), so the remote side returns the original result when it sees the key again. If it doesn’t, store a stable external ID on the remote ticket (ServiceNow’s incident sys_id is ideal) and, before any retry, look it up.
This Node.js simulation (plain JavaScript; the “remote” is a fake whose first reply is lost after it has already created the ticket, and whose second call fails with 503) compares a naive retry with both techniques:
// A remote ticketing API whose first reply is lost after it has already created the ticket.
function makeRemote({ honoursIdempotencyKey }) {
const tickets = [], seenKeys = new Map();
const script = ['processed-but-reply-lost', 'unavailable-503', 'ok']; // what happens on calls 1, 2, 3...
let call = 0;
return {
tickets,
post(body, headers = {}) {
const behaviour = script[Math.min(call++, script.length - 1)];
if (behaviour === 'unavailable-503') return { status: 503 };
const key = headers['Idempotency-Key'];
let ticket = honoursIdempotencyKey && key ? seenKeys.get(key) : undefined;
if (!ticket) { ticket = { id: `VND-${700 + tickets.length}`, ...body }; tickets.push(ticket); if (key) seenKeys.set(key, ticket); }
if (behaviour === 'processed-but-reply-lost') return { status: 0, error: 'Read timed out' };
return { status: 201, body: ticket };
},
findByExternalId(id) { return tickets.find((t) => t.external_id === id); },
};
}
function classify(res) {
if (res.status === 0) return 'unknown'; // timeout or connection reset: it may have worked
if (res.status === 429 || res.status >= 500) return 'retry';
if (res.status >= 200 && res.status < 300) return 'done';
return 'fail'; // other 4xx: retrying won't help
}
function createWithRetries(remote, incident, { sendKey, lookupBeforeRetry }) {
const body = { external_id: incident.sys_id, summary: incident.short_description };
for (let attempt = 1; attempt <= 5; attempt++) {
if (lookupBeforeRetry && attempt > 1) {
const existing = remote.findByExternalId(incident.sys_id);
if (existing) return `attempt ${attempt}: found ${existing.id} by external_id, no new POST`;
}
const res = remote.post(body, sendKey ? { 'Idempotency-Key': `create-${incident.sys_id}` } : {});
const verdict = classify(res);
if (verdict === 'done') return `attempt ${attempt}: created ${res.body.id}`;
if (verdict === 'fail') return `attempt ${attempt}: gave up (${res.status})`;
}
return 'exhausted retries';
}
const incident = { sys_id: '9d385017c611228701d22104cc95c371', short_description: 'Payroll portal down' };
for (const [label, remoteOpts, clientOpts] of [
['naive retry', { honoursIdempotencyKey: false }, {}],
['Idempotency-Key header', { honoursIdempotencyKey: true }, { sendKey: true }],
['look up by external_id', { honoursIdempotencyKey: false }, { lookupBeforeRetry: true }],
]) {
const remote = makeRemote(remoteOpts);
const result = createWithRetries(remote, incident, clientOpts);
console.log(`${label.padEnd(24)} ${result.padEnd(54)} remote tickets: ${remote.tickets.map((t) => t.id).join(', ')}`);
}
// Exponential backoff ceilings (wait a random time between 0 and the ceiling: "full jitter")
const base = 30, cap = 900;
console.log('backoff ceilings (s):', [1, 2, 3, 4, 5, 6].map((n) => Math.min(cap, base * 2 ** (n - 1))).join(', '));naive retry attempt 3: created VND-701 remote tickets: VND-700, VND-701
Idempotency-Key header attempt 3: created VND-700 remote tickets: VND-700
look up by external_id attempt 2: found VND-700 by external_id, no new POST remote tickets: VND-700
backoff ceilings (s): 30, 60, 120, 240, 480, 900The naive client did everything a reasonable person would: it saw a timeout, retried, got a 503, retried again, and succeeded. The vendor now has two tickets for one incident, VND-700 (from the lost reply) and VND-701. With an idempotency key, the third attempt carries the same key as the first, and the remote returns VND-700 instead of creating a new one. With a lookup by external ID, the client discovers on its second attempt that the first one worked and stops.
The key must be stable for the logical operation. A random UUID per attempt defeats the purpose; create-<incident sys_id> or an operation ID stored on the outbox row works. Once you have the remote ID, store it locally, for example in the task’s correlation_id and correlation_display fields, so updates can target the right ticket and a later “create” can be skipped.
Classifying the response
Not every failure deserves a retry. A useful split:
- 2xx: done. Record the remote ID.
- 429 Too Many Requests and 5xx: transient. Retry later, honouring a
Retry-Afterheader if present. - Other 4xx (400, 401, 403, 404, 422): retrying the same request will fail the same way. A bad payload, expired credentials or a missing remote object needs a person or a code fix, so mark the row failed and route it to a queue someone watches.
- No status (timeout, reset): unknown. Retry only through an idempotent path.
A 409 Conflict deserves its own handling if the remote uses it to say “this already exists”: treat it as success and fetch the existing ID.
Backoff and giving up
Retrying immediately, in a tight loop, hits a struggling remote system exactly when it can least cope, and when many records fail together, every retry arrives at once. Use exponential backoff with a cap and random jitter. With a 30-second base and a 15-minute cap, the ceilings are those in the last line of the output above: 30, 60, 120, 240, 480 and 900 seconds. “Full jitter” waits a random time between zero and the ceiling, which spreads a burst of failures out.
Then stop. After a fixed number of attempts or a total time budget, mark the operation failed, keep the payload and the last error, and create something a human will see: an incident for the integration team, or a row in a monitored list. Silent exhaustion is the worst outcome, because everyone assumes the ticket exists.
Writing it on the platform
An illustrative client script include (not run on an instance) that a scheduled job processing the outbox could call. It sends the idempotency key, sets a timeout and turns the response into one of the outcomes above, leaving scheduling decisions to the caller:
var VendorTicketClient = Class.create();
VendorTicketClient.prototype = {
initialize: function() {},
createTicket: function(incident, operationId) {
var r = new sn_ws.RESTMessageV2('Vendor Ticketing', 'create_ticket');
r.setHttpTimeout(10000); // milliseconds
r.setRequestHeader('Idempotency-Key', operationId);
r.setRequestBody(JSON.stringify({
external_id: incident.getUniqueValue(),
summary: incident.getValue('short_description')
}));
var resp = r.execute();
var status = resp.getStatusCode();
if (resp.haveError() && !status) return { outcome: 'unknown', detail: resp.getErrorMessage() };
if (status == 200 || status == 201) return { outcome: 'done', remoteId: JSON.parse(resp.getBody()).id };
if (status == 409) return { outcome: 'exists', detail: resp.getBody() };
if (status == 429 || status >= 500) return { outcome: 'retry', retryAfter: resp.getHeader('Retry-After') };
return { outcome: 'failed', detail: status + ' ' + resp.getBody() };
},
type: 'VendorTicketClient'
};Integration Hub spokes and Flow Designer REST steps cover a lot of this declaratively, including retry settings on REST steps in recent releases. The design questions don’t go away: you still decide which operations are safe to retry and what a person sees when retries run out.
Inbound calls need the same care
The same problem arrives from the other direction. A partner’s system will retry when your response is slow, so an inbound “create incident” should look for an existing record with the partner’s ID in correlation_id before inserting, and return the existing record rather than a second one. Scripted REST APIs make that straightforward: check, then insert, and return 200 with the existing record (or 409, if that’s the agreed contract) on a repeat. Put a unique index or a before rule behind it if two requests can arrive at the same moment.
Interview practice
An after-insert business rule on incident calls a vendor’s API with RESTMessageV2 to open a matching ticket. Users complain that saving incidents sometimes takes 30 seconds, and the vendor reports duplicate tickets. What would you change?
Answer
The slow saves happen because the call runs inside the user’s transaction: when the vendor is slow, the save waits, up to whatever timeout applies. The duplicates are the lost-reply problem: the vendor created the ticket, the reply didn’t arrive in time, and something (a user saving again, or retry logic) sent another create.
I’d move the call out of the transaction. The business rule only writes an outbox row (or queues an event) with an operation ID such as create-<sys_id>. A background job sends the request with an explicit timeout and an Idempotency-Key built from that ID, if the vendor supports it; otherwise it looks the ticket up by external ID before every retry. Responses are classified: 2xx stores the vendor ID in correlation_id, 429 and 5xx back off exponentially with jitter, other 4xx stop and alert, and timeouts retry through the idempotent path. After a fixed number of attempts the row is marked failed and an incident is raised for the integration owners. For the existing duplicates, I’d reconcile by external ID with the vendor and close the extras.
Related reading
- ServiceNow interview questions
- ServiceNow multiple-choice questions
- ServiceNow REST API Design, for the inbound side of the contract
- ServiceNow Business Rule Timing and Side Effects, on why calls don’t belong in before or after rules
- RESTMessageV2 API (ServiceNow documentation)