Skip to content

LDAP Connector Values Already in Use

A directory often holds values JIM is about to write: the same value already in place on the object, or a unique value already held by another. This page covers how the LDAP Connector treats each.

How an export lands when a value is already in place JIM sends a modify, attaching the Permissive Modify control where the directory advertises it, and an accepted modify is done. When the directory refuses it because a value is already in place, JIM applies it again in parts, one kind of change at a time: object classes added, replacements, values removed and re-added under a different case, values added, values removed, object classes removed. When the directory refuses a part, JIM halves it, and halves again only the halves refused, until each refused value stands alone. That value counts as done, and every other change in the export lands. A refusal for any other reason fails the export, which is retried later. refused: in place part refused Send the modifyPermissive Modifywhere advertised Apply in partsone kind ofchange at a time Halve the partuntil each refusedvalue stands alone Every otherchange lands accepted: done a refused value counts as done How an export lands when a value is already in place JIM sends a modify, attaching the Permissive Modify control where the directory advertises it, and an accepted modify is done. When the directory refuses it because a value is already in place, JIM applies it again in parts, one kind of change at a time: object classes added, replacements, values removed and re-added under a different case, values added, values removed, object classes removed. When the directory refuses a part, JIM halves it, and halves again only the halves refused, until each refused value stands alone. That value counts as done, and every other change in the export lands. A refusal for any other reason fails the export, which is retried later. refused: in place part refused Send the modifyPermissive Modify where advertisedaccepted: done Apply in partsone kind of change at a time Halve the partuntil each refused value stands alone Every other change landsa refused value counts as done

Values already in place

An export asks the directory for a state, so adding a member who is already in the group, or removing one who has already left, is a change that is already done. A directory sees it differently: an LDAP modify is all or nothing, so a directory that finds one value already present (or already absent) refuses the whole request, along with every other change it carried. JIM makes sure those other changes still land:

  • Where the directory offers Permissive Modify
    When JIM opens the export connection it reads the directory's rootDSE, and where the directory advertises the Permissive Modify control (1.2.840.113556.1.4.1413), JIM attaches it to every modify that adds or removes values, including the extra modifies that carry a large group's members beyond the Modify Batch Size. The directory then accepts a value already in place as a success. Active Directory and Samba AD advertise it. JIM never sends the control to a directory that does not advertise it, because a directory that does not understand it refuses the whole modify.
  • Where it does not, or the directory refuses anyway
    JIM applies the refused modify again in parts: any object classes being added first, then the attributes being replaced (together, in one request), then any value removed and added again under a different case (see below), then the other values being added, then those being removed, then any object classes being removed. Each attribute's adds go as one request, and its removals as another. Where the directory refuses a part because a value in it is already in place, JIM splits that part in half and sends each half, splitting again only the halves the directory refuses again, until each value it refuses stands alone; that value counts as done, and every other value is applied. OpenLDAP and 389 Directory Server do not advertise the control, and Samba AD still refuses to remove a member the group does not hold even with it, so for those this is the path.

Halving keeps the cost down on large groups. One member already in place among the Modify Batch Size's 1,000 takes about 20 requests rather than 1,000, and only about 10 of them change the group. That matters most on OpenLDAP, where every change to a group rewrites the whole entry, and logs its entire member list too if the accesslog records old values (logold), which JIM does not need.

One kind of add needs more than the directory's "already present" before JIM counts it as done: a value removed and added again under a different case. Active Directory compares some attributes without regard to case, proxyAddresses among them, so when an address changes from primary (SMTP:jane.doe@example.com) to alias (smtp:jane.doe@example.com), JIM removes one spelling and adds the other. Added first, the new spelling would be skipped as already present, or accepted where the old one has already gone, and the removal would then take the address away. So JIM sends the removal and the add in one request, removal first, whether the Modify Batch Size divides the change or a modify is applied in parts. If the directory refuses that request because a value in it is already in place, JIM sends the removal, then the add. Any other add the directory reports as already present while removals of that attribute are still to come is sent again after them, and counts as done only if reported present again.

A single-valued attribute never needs this. JIM replaces its value in one modification, whatever the plurality of the Metaverse attribute an Attribute Flow takes it from, and a directory never refuses a replacement as already present.

Nothing is lost either way, and JIM never reports a refused modify as a success. If the directory refuses any value for another reason (insufficient access rights, say), the export fails with the directory's own message and JIM retries the Pending Export on later export runs, as with any failed export; values already applied by then count as done on the retry. The worker log records each modify applied in parts, with how many requests it took and how many of its value changes were already in place, and the end of each export run records the total.

JIM recognises "already present" from attributeOrValueExists (20) and "already absent" from noSuchAttribute (16) on every directory. Active Directory and Samba AD report group membership differently, so on those JIM also recognises entryAlreadyExists (68) when adding a member, and unwillingToPerform (53) when removing one, but only where the directory says the member is not in the group (Active Directory's 00000561 code, or Samba AD's "doesn't exist for target GUID" or, for a member the group held once, "already deleted for target GUID"). unwillingToPerform for any other reason, such as removing a member from its primary group, still fails the export.

Probing for values already in use

When a synchronisation generates a value that is exported unchanged to an attribute of this directory, such as an account name sent to sAMAccountName or uid, JIM asks the directory whether the value is already in use before choosing it. This catches accounts JIM does not import: one outside the containers in the Connected System's scope, or one created by hand. How generated values are checked, and what is and is not probed, is in Checking availability in target systems.

  • One connection per synchronisation
    The connection is opened with this Connected System's own settings, credentials and certificates, the first time the synchronisation needs it, and closed when the run ends.
  • Searched from the partition root
    Each selected partition is searched from its root down, not only the containers JIM imports, because an account outside the import scope is exactly what the probe exists to find.
  • One search per object
    The object's candidates (up to ten) and a control value JIM already holds go in one search covering every attribute JIM probes here, each answered from its own values. If together they match more entries than a probe allows, or the search is refused, JIM searches each attribute alone. A control value that does not come back means the account JIM binds as cannot see that attribute, so JIM stops trusting the probe for it this run.
  • Email addresses include aliases
    In Active Directory and Samba AD, a probe of mail also looks for the address in every object's proxyAddresses, where Exchange keeps each address a mailbox, group or contact receives mail at (SMTP: for the primary address, smtp: for the rest). An address held as someone else's alias is in use, so JIM chooses another. Entries of other kinds, such as SIP: and X500:, are not email addresses and do not count. In other directories, only mail is searched. JIM's own records are checked against mail only; only the directory search finds aliases.
  • Naming attributes are not probed
    The Distinguished Name, and the attributes that name entries (cn, ou, o, dc, c, l, st), only need to be unique within their container, so a directory-wide search would report false clashes. uid is probed.
  • Undetermined is not a failure
    A directory that cannot be reached, refuses the search, times out or returns more entries than a probe allows is reported once on the synchronisation's Activity as a warning naming the reason, and the run continues with JIM's own records for that Connected System.

Forest-wide attributes and the Global Catalog

Some Active Directory attributes must be unique across the whole forest, not just one domain: userPrincipalName and servicePrincipalName, which Active Directory itself keeps unique forest-wide, and mail and proxyAddresses, because two accounts anywhere in one forest with the same address is a clash for mail routing and for anything that synchronises the forest onward. Searching only this Connected System's own domain cannot see an account in another domain of the forest that already holds the value, so for these four attributes JIM searches the forest through a Global Catalog instead: one search across every domain for whichever it is checking.

  • When the Global Catalog is used
    Always, when the Global Catalog Server setting names one. When it is blank, JIM counts the forest's domains: with one domain, the domain is the forest and JIM searches it as for any other attribute; with more than one, JIM uses the domain controller it connects to, if that domain controller is a Global Catalog (most are).
  • Connection
    Port 3268, or 3269 when Use Secure Connection (LDAPS)? is on, with this Connected System's own credentials and certificates. Allow that port from JIM to the Global Catalog through any firewall between them. The connection is opened the first time a synchronisation probes one of these attributes, and closed when the run ends.
  • Replicated attributes only
    A Global Catalog holds only the attributes in the forest's partial attribute set. All four are in it by default; if one has been removed, JIM searches the domain for it instead.
  • The control value applies here too
    A Global Catalog that does not return a value JIM knows is in this very domain cannot be trusted for that attribute, and JIM searches the domain for that attribute alone.
  • When the Global Catalog cannot be searched
    If the forest has more than one domain and no Global Catalog is available, or the Global Catalog cannot be reached (JIM waits no longer than the Connection Timeout, and at most 10 seconds; where its name resolves to several addresses, JIM gives up at the first that does not answer rather than going on to the next), refuses the search or cannot see the attribute, JIM still searches this Connected System's own domain, so a clash there is still caught, and records a warning on the synchronisation's Activity naming the attribute and what kept the Global Catalog out of reach. Nothing is lost: the values JIM chose are as safe as they were before forest-wide probing, and if one is in use in another domain, Active Directory refuses the export and JIM corrects it (see When a target rejects a value). Fix the cause (open the port, set Global Catalog Server to a Global Catalog JIM can reach, or make the domain controller JIM connects to a Global Catalog) and the next run searches the forest.

Other attributes, and every attribute in a directory that is not Active Directory or Samba AD (where mail is an ordinary attribute), are searched in the partition as described above. Samba AD's Global Catalog is used on the same terms, though a Samba AD forest has only one domain.