ServiceNow CMDB Identification and Data Quality

A CMDB is only useful if people trust it. When a change manager looks up the impact of patching web01 and finds three records with different owners and RAM figures, they stop using the CMDB and start asking in chat. Duplicates are rarely caused by one big mistake. They come from several data sources describing the same machine slightly differently, and a load process that can’t tell they mean the same thing. Interviewers asking about CMDB quality want to hear how you stop that at the door rather than clean it up every quarter.

How one server becomes three records

Imagine three feeds describing the same virtual machine:

• Discovery reports web01.acme.local with serial VMware-42 1a 7f 9c, 16,384 MB of RAM.
• SCCM (through an import) reports WEB01 with serial vmware-421a7f9c and 16,000 MB.
• A manual CSV from the web team says web01, no serial, support group “Web Platform”.

If each feed is loaded with a transform map that coalesces on name, you get three CIs, because web01.acme.local, WEB01 and web01 are different strings. If you coalesce on serial number, the manual record still creates a CI of its own (it has no serial), and the two serials only match if someone normalises case and spacing. Every downstream process now picks a CI more or less at random.

Identification: is this CI new?

ServiceNow’s answer is the Identification and Reconciliation Engine (IRE). For identification, each CI class has an identification rule made of one or more identifier entries, each with a priority. An entry names the attributes that uniquely identify a CI, for example serial number, or name, or a combination. The engine tries the entries in priority order, strongest first, and stops at the first one the payload can satisfy and that finds a match. If no entry matches, the CI is inserted.

A few details from the documentation come up in design reviews:

• Derivation. A child class with no rule of its own derives its parent’s rule. As soon as you define a rule on the child, the parent’s rule no longer applies to it, including its related entries, so copy what you still need.
• Lookup entries. Some identifiers live in a separate table that references the CI, such as cmdb_serial_number or network adapters. Lookup-based entries let identification use them.
• Dependent CIs. An application CI can’t be identified on its own; it’s identified in the context of the server hosting it. Dependent identification rules identify the parent first.

Choosing the attributes is the real design work. Serial numbers are strong for physical hardware but can be duplicated by cloning templates. Names are weak (renames, reuse) but often the only shared value. Cloud resources usually have a provider object ID that beats both.

Reconciliation: who may change what?

Once the engine knows which CI a payload refers to, reconciliation rules decide which attributes that source may update. Static reconciliation rules list the discovery sources authorised to write a class’s attributes, in priority order. If Discovery outranks SCCM for RAM, an SCCM value of 16,000 doesn’t overwrite Discovery’s 16,384. A source with no rule for an attribute can’t write it at all. Child class rules override parent class rules, and data refresh rules let a lower-priority source take over when the higher one’s data goes stale. Newer releases also offer dynamic reconciliation rules, which pick values based on data across sources (they need CMDB 360) and take precedence over static rules for the same attribute.

A small model of both steps

This Node.js sketch (illustrative logic, not the IRE itself) runs the three payloads through naive name coalescing and then through prioritised identifiers, normalisation and static reconciliation:

const payloads = [
  { source: 'ServiceNow', values: { name: 'web01.acme.local', serial_number: 'VMware-42 1a 7f 9c', ram: 16384, os: 'Linux' } },
  { source: 'SCCM',       values: { name: 'WEB01', serial_number: 'vmware-421a7f9c', ram: 16000, os: 'Linux' } },
  { source: 'ManualEntry', values: { name: 'web01', support_group: 'Web Platform', ram: 8192 } },
];

// Normalisation is part of identification: compare values in a canonical form.
const norm = {
  serial_number: (s) => s && s.toLowerCase().replace(/[^a-z0-9]/g, ''),
  name: (n) => n && n.toLowerCase().split('.')[0],
};
// Identifier entries, strongest first.
const identifiers = [['serial_number'], ['name']];
// Static reconciliation: which sources may write an attribute, in priority order.
const authority = { ram: ['ServiceNow', 'SCCM'], os: ['ServiceNow', 'SCCM'], support_group: ['ManualEntry'] };

function naive(cis, p) {                                  // coalesce on the raw name only
  let ci = cis.find((c) => c.name === p.values.name);
  if (!ci) cis.push((ci = { ...p.values }));
  return cis;
}

function ire(cis, p) {
  let ci, how = 'inserted';
  for (const entry of identifiers) {
    if (!entry.every((f) => p.values[f])) continue;        // payload lacks this identifier
    ci = cis.find((c) => entry.every((f) => norm[f](c[f]) === norm[f](p.values[f])));
    if (ci) { how = `matched on ${entry.join('+')}`; break; }
  }
  if (!ci) { cis.push((ci = { sys_id: `ci${cis.length + 1}`, name: p.values.name, owner: {} })); }
  const notes = [];
  for (const [field, value] of Object.entries(p.values)) {
    const allowed = authority[field];
    if (!allowed) { ci[field] ??= value; continue; }        // identifiers and unlisted fields: set if empty
    const rank = allowed.indexOf(p.source);
    const current = ci.owner[field] === undefined ? Infinity : allowed.indexOf(ci.owner[field]);
    if (rank === -1) notes.push(`${field} refused`);
    else if (rank <= current) { ci[field] = value; ci.owner[field] = p.source; }
    else notes.push(`${field} kept from ${ci.owner[field]}`);
  }
  console.log(`  ${p.source.padEnd(11)} ${how.padEnd(24)} ${notes.join(', ')}`);
  return cis;
}

console.log('naive coalesce on name ->', payloads.reduce(naive, []).length, 'CIs');
console.log('identification rules:');
const cis = payloads.reduce(ire, []);
console.log('->', cis.length, 'CI:', JSON.stringify({ ...cis[0], owner: undefined }));

naive coalesce on name -> 3 CIs
identification rules:
  ServiceNow  inserted                 
  SCCM        matched on serial_number ram kept from ServiceNow, os kept from ServiceNow
  ManualEntry matched on name          ram refused
-> 1 CI: {"sys_id":"ci1","name":"web01.acme.local","serial_number":"VMware-42 1a 7f 9c","ram":16384,"os":"Linux","support_group":"Web Platform"}

The SCCM payload matches on serial only because both serials were normalised to vmware421a7f9c; without that one line, identification falls through to the name entry and still matches, but only because names were normalised too. The manual payload has no serial, so the engine skips that entry and matches on name. Its RAM figure of 8,192 (a typo, or an old spec sheet) is refused because the manual source has no authority over RAM, while its support group is accepted because it’s the only source allowed to set it. One CI, and each attribute has a known owner.

The model also shows the risk in a weak identifier: if a second machine called web01 existed in another domain, name matching would merge them. That’s why name entries sit at the bottom of the priority list and are often combined with another attribute.

Feeding the CMDB through IRE

The engine only protects data that goes through it. On the platform, integrations should call the IRE API rather than inserting CIs with GlideRecord. ServiceNow’s API documentation says as much: use it instead of updating the CMDB directly. An illustrative scoped call (not run here):

var payload = {
    items: [{
        className: 'cmdb_ci_linux_server',
        values: { name: 'WEB01', serial_number: 'vmware-421a7f9c', ram: '16000', os: 'Linux' }
    }]
};
var result = sn_cmdb.IdentificationEngine.createOrUpdateCI('SCCM', JSON.stringify(payload));
gs.info(result);   // JSON describing, per item, what IRE did and which CI it touched

The source name must be one of the choice values on the discovery_source field of cmdb_ci, so register new sources there and give each one reconciliation rules before it goes live. For spreadsheet-style loads, use an import path that routes through IRE instead of a transform map that coalesces straight into a CI table. Manual edits in the form are a source too (ManualEntry), and reconciliation rules apply to it.

Measuring quality without counting records

A growing CI count isn’t progress. CMDB Health reports four KPIs, and they’re a better vocabulary for an interview answer:

• Completeness: required and recommended fields populated.
• Correctness: duplicates (via identification rules), orphans and stale CIs.
• Compliance: CIs audited against defined certificates.
• Relationships: orphan and duplicate relationships, and expected hosting or containment relationships.

Apply them where decisions are made: the classes that change, incident and impact analysis depend on (servers, databases, application services), not every class that a discovery tool can fill. Give each of those classes an owning group, a staleness threshold that matches how often its sources report, and a de-duplication task process for the matches the engine couldn’t resolve on its own.

Interview practice

After a new cloud integration went live, the count of Linux server CIs doubled overnight. Spot checks show pairs of CIs with the same hostname: one from Discovery, one from the cloud feed. The cloud feed sends a provider instance ID, hostname and IP address. What went wrong, and how would you fix it without losing data?

Answer

The identification rule for the class didn’t have an entry the two sources could both satisfy. Discovery’s records carry serial number and name; the cloud feed probably sent no serial, and if its hostnames are short names while Discovery stores FQDNs, a name entry didn’t match either. Or the integration bypassed IRE and inserted directly, in which case no rule was consulted.

First I’d stop the bleeding: pause the integration and confirm whether it calls the IRE API with a registered source. Then I’d fix identification: add a strong entry for the cloud object ID (populating it on the Discovery records where the mapping is known), make sure name comparisons normalise case and domain suffix, and keep name low in priority. Add reconciliation rules so the cloud feed owns cloud-specific attributes and Discovery keeps OS-level ones. For the existing pairs, use de-duplication tasks to merge each pair into the older CI, which keeps the relationships, incidents and changes already pointing at it. Finally, rerun the feed in a sub-production instance and confirm the count stays flat and the IRE output reports updates, not inserts.

Related reading

• ServiceNow interview questions
• ServiceNow multiple-choice questions
• ServiceNow REST API Design, for the integration side of CI feeds
• ServiceNow Scheduled Jobs, which run the imports and health jobs
• Identification rules (ServiceNow documentation)