Skip to content

LDAP Connector Delta Import

A Delta Import reads only what changed since the last import, from wherever the directory records its changes. Each directory type keeps that record differently, and each Delta Import fails safe rather than miss a change: it stops and asks for a Full Import when it cannot be sure it would see them all.

Where each directory type records its changes Active Directory and Samba AD: a Delta Import reads objects whose uSNChanged has passed the watermark, against the domain controller JIM pinned, and finds deletions by a tombstone search of the Deleted Objects container. It fails fast when the domain controller changed or was restored. OpenLDAP: it reads cn=accesslog, only the selected partitions' writes, and matches each deletion to its object by reqEntryUUID. It fails fast when the overlay's purge, or a restore, has removed the entry its watermark names. 389 Directory Server and generic directories: it reads the changelog the Root DSE advertises, otherwise cn=changelog, and identifies a deletion from the copy of the deleted entry the changelog keeps, which 389 Directory Server keeps only with nsslapd-log-deleted on. It fails fast when the changelog no longer continues from the watermark, because it was trimmed or the directory restored. Whatever the type, before it reads a change a Delta Import confirms that the account JIM connects as can read the source, and a run that fails fast asks for a Full Import. Active Directory and Samba AD OpenLDAP the accesslog overlay 389 and generic changelog directories Reads changes from uSNChangedon the pinneddomain controller cn=accesslogthe selected partitions' writes the changelogthe Root DSE's, elsecn=changelog Finds deletions by a tombstone searchof the DeletedObjects container reqEntryUUIDlogged with every deletion the deleted entry's copy389: needsnsslapd-log-deleted on Fails fast when the domaincontroller changedor was restored the purge passedthe watermarkor a restore removed it the changelog nolonger continuestrimmed, or thedirectory restored Every type: before it reads a change, a Delta Import confirms the account JIM connects as can read the source Where each directory type records its changes Active Directory and Samba AD: a Delta Import reads objects whose uSNChanged has passed the watermark, against the domain controller JIM pinned, and finds deletions by a tombstone search of the Deleted Objects container. It fails fast when the domain controller changed or was restored. OpenLDAP: it reads cn=accesslog, only the selected partitions' writes, and matches each deletion to its object by reqEntryUUID. It fails fast when the overlay's purge, or a restore, has removed the entry its watermark names. 389 Directory Server and generic directories: it reads the changelog the Root DSE advertises, otherwise cn=changelog, and identifies a deletion from the copy of the deleted entry the changelog keeps, which 389 Directory Server keeps only with nsslapd-log-deleted on. It fails fast when the changelog no longer continues from the watermark, because it was trimmed or the directory restored. Whatever the type, before it reads a change a Delta Import confirms that the account JIM connects as can read the source, and a run that fails fast asks for a Full Import. Active Directory and Samba AD Reads changes fromuSNChangedon the pinned domain controller Finds deletions bya tombstone searchof the Deleted Objects container Fails fast whenthe domain controller changedor was restored OpenLDAP the accesslog overlay Reads changes fromcn=accesslogthe selected partitions' writes Finds deletions byreqEntryUUIDlogged with every deletion Fails fast whenthe purge passed the watermarkor a restore removed it 389 and generic changelog directories Reads changes fromthe changelogthe Root DSE's, else cn=changelog Finds deletions bythe deleted entry's copy389: needs nsslapd-log-deleted on Fails fast whenthe changelog no longer continuestrimmed, or the directory restored Every type: before it reads a change, a Delta Import confirms the account JIM connects as can read the source

Active Directory

Uses USN (Update Sequence Number) change tracking. USNs are only meaningful when read back against the same domain controller that issued them, so JIM also records the domain controller's identity (its invocationId, falling back to its hostname where an invocationId is not available for comparison) and verifies it on every Delta Import before querying for changes. If the pinned domain controller changed since the last run, or was restored from a backup or snapshot, the Delta Import fails fast with an error naming what changed, before it reads a single change, rather than silently skipping or re-importing changes. See Domain Controller Discovery and Pinning and Delta import fails with a domain controller mismatch error below. JIM has not been tested against a domain with the Active Directory Recycle Bin enabled.

OpenLDAP

Uses the accesslog overlay (cn=accesslog). The accesslog is shared by every database on the server, so a Delta Import asks the directory for the selected partitions' writes only, reads them a page at a time (the Run Profile's page size), and matches a deletion to the object JIM holds by the deleted entry's entryUUID, which the accesslog records with every deletion (reqEntryUUID); its memory stays bounded however much was written. JIM never reads the old values the overlay can log (logold, logoldattr), so leave them off: logold logs the old values of every attribute a write changes, and on a large group that is the whole member list on every membership change, often many times the size of the directory itself. A deletion the accesslog records without reqEntryUUID cannot be matched; the run carries a warning naming it, and a Full Import detects it by absence. The account JIM connects as needs the size limit lifted on the accesslog database as well as on each suffix: a Delta Import the limit stops part-way ends as Failed with error naming the limit, rather than carry on from a point that could miss a change. The overlay's purge (olcAccessLogPurge) discards entries older than its maximum age, so keep that age longer than the longest gap between Delta Imports. Before it reads a single change, a Delta Import reads back the accesslog entry its watermark names; if the purge, or a restore of the directory, has removed it, the Delta Import fails fast and asks for a Full Import rather than silently missing the changes discarded with it, and the watermark is left as it was. If nothing at all was written in that time nothing was lost, but JIM cannot tell, so a directory left without writes for longer than the purge age needs one Full Import before its Delta Imports resume.

389 Directory Server and generic directories

Uses the directory's changelog: the one its root DSE advertises (the changelog attribute, together with firstChangeNumber and lastChangeNumber), otherwise cn=changelog. A Delta Import reads it a page at a time (the Run Profile's page size), each page carrying on from the exact change number the last ended at, and reads a deleted entry's copy only for a deletion, so its memory stays bounded however much was written. Where the root DSE advertises the last change number, JIM takes its watermark from it rather than reading the changelog to find it. An added or changed object is read as it now stands; a renamed object is followed to its new name; a deleted object, which can no longer be read, is identified from the copy of the deleted entry the changelog keeps with the deletion. On 389 Directory Server that copy is kept only while the Retro Changelog plug-in's nsslapd-log-deleted setting is on (it is off by default), so JIM checks the setting: Schema Discovery warns while it is off, and a Delta Import refuses to run rather than import additions and updates while missing every deletion. Nothing is lost by the refusal: Full Import is unaffected and detects deletions by absence, and Delta Import runs once the setting is on and the directory restarted. Where JIM cannot read the plug-in's settings, it warns that it could not confirm and runs. A deletion whose changelog entry carries no copy of the deleted entry is reported as a warning on the run, naming the object, which stays in JIM until a Full Import. Because a deletion's entry carries the deleted object's attributes, exclude userPassword from the changelog (dsconf <instance> plugin retro-changelog set --exclude-attrs userPassword). 389 Directory Server publishes a changelog only once its Retro Changelog plug-in is enabled, and grants read on it to nobody but the Directory Manager by default; the grants, and the plug-in settings, are under 389 Directory Server. If the changelog has been trimmed past the last import's watermark (the plug-in's maximum age), the Delta Import fails fast naming the gap rather than silently skipping the trimmed changes, so keep the maximum age longer than the longest gap between Delta Imports. A Delta Import also confirms that the changelog still continues from the last import's watermark, and otherwise fails fast naming what it found and asks for a Full Import, leaving the watermark as it was. It refuses when the changelog's newest change number (the root DSE's lastChangeNumber) is below the watermark, which is how 389 Directory Server looks just after a restore from files or a snapshot, since it then numbers the next changes from the restored point; when the watermark's own change now carries a different changeTime, which is the same restore once enough writes have carried the numbering past the watermark again; and when the watermark's own change is gone while older ones remain, which is what an online restore (dsconf <instance> backup restore) leaves, since it takes the changes made after the backup out of the changelog and carries on numbering after them. Each also catches Host reaching a different server, since each server numbers its own changelog. A directory that advertises neither where its changelog starts (firstChangeNumber) nor when each change was made (changeTime) can only be checked by number, so after restoring one, run a Full Import.

Delta Import change source checks

A Delta Import reads what changed from wherever the directory keeps that record: the domain's Deleted Objects container for deletions in Active Directory and Samba AD (changes themselves come from uSNChanged), cn=accesslog for OpenLDAP, and the changelog for 389 Directory Server and generic directories. None of these refuses an account that lacks rights over it in a way JIM can rely on: a search can return nothing. So before it queries any change, the run confirms that the account JIM connects as can read the source; the same check runs at Schema Discovery and on the schema refresh preview, where its findings are warnings.

  • When the source is provably unreadable (the account is not allowed to list the Deleted Objects container; the accesslog or changelog does not exist, or the directory refuses to read it), the run ends as Failed with error before importing anything, and its message names the source, the consequence and the remedy. Continuing would have imported what the account could see and provably missed the rest.
  • When JIM could not confirm either way (most often because the account lacks Read Permissions on the Deleted Objects container, or the connection failed during the check), the run goes ahead and completes as Complete with warning, saying what it could not confirm and why, for example:

JIM could not confirm that the account it connects as can list the Deleted Objects container (CN=Deleted Objects,DC=corp,DC=local): the directory did not return the container's permissions, which needs Read Permissions on it. If it cannot, Delta Imports from this domain import no deletions. Granting Read Permissions on that container alongside List Contents and Read Property lets JIM give a definite answer.

  • When the last import recorded no watermark (its change source could not be read at the time, or the record predates this version of JIM), the run performs a Full Import instead, completes as Complete with warning saying so, and records the watermark the next Delta Import reads from. This is the same recovery the SQL and SCIM Connectors make.
  • An Active Directory run also completes as Complete with warning when the tombstone search itself is refused or finds no container; that warning begins "Deletions were not detected in " and notes that objects deleted since the last import may still be present in JIM, with one note per domain. For OpenLDAP and changelog directories there is no such half-way: a refused search of the one log the run reads from ends the run as Failed with error.
  • A changelog directory whose changelog has been trimmed past the last import's watermark (389 Directory Server: the Retro Changelog plug-in's maximum age) also ends as Failed with error, naming the first change number the changelog still holds and the one the last import ended at; a Full Import re-establishes the baseline.
  • On a changelog directory the check has a second part, because a deletion cannot be read back and is identified from the copy of the deleted entry the changelog keeps with it. On 389 Directory Server that copy is kept only while the Retro Changelog plug-in's nsslapd-log-deleted setting is on, so JIM reads the plug-in's entry (cn=Retro Changelog Plugin,cn=plugins,cn=config) as well as the changelog. Readable and on, and the run proceeds. Readable and off, and the run ends as Failed with error naming the setting to turn on and that a Full Import detects deletions by absence meanwhile; Schema Discovery warns with the same message. Not readable (the account may not read the entry, or the directory is not 389 Directory Server and has no such entry), and the run goes ahead and completes as Complete with warning, saying JIM could not confirm that deletions will be detected and which read to grant. A deletion whose changelog entry carries no copy of the deleted entry is not staged; the run carries a warning naming the object, which stays in JIM until a Full Import. The grants for both reads are under 389 Directory Server.

The grants that make each answer definite are under Service Account Permissions.