Pending Export Lifecycle¶

Last updated: 2026-07-21, JIM v0.13.0

This diagram shows the full lifecycle of a Pending Export from creation during synchronisation, through export execution, to confirmation during a confirming import. Pending Exports are the mechanism by which JIM propagates changes from the metaverse to target Connected Systems.

State Diagram¶

stateDiagram-v2
    [*] --> Pending: Created during Sync<br/>(export evaluation)

    Pending --> Executing: Export run starts<br/>batch marked executing

    Executing --> Exported: Connector reports<br/>success

    Executing --> ExportNotConfirmed: Connector reports<br/>failure (retryable)
    Executing --> Failed: ErrorCount >= MaxRetries

    Exported --> [*]: Confirming import confirms<br/>all attribute values match<br/>(PE deleted)

    Exported --> ExportNotConfirmed: Confirming import finds<br/>attribute values don't match

    ExportNotConfirmed --> Executing: Next export run<br/>(after NextRetryAt backoff)

    ExportNotConfirmed --> Pending: Sync re-evaluates<br/>and reasserts changes

    ExportNotConfirmed --> Failed: ErrorCount >= MaxRetries<br/>(permanent failure)

    Failed --> [*]: Manual intervention<br/>or PE deleted

    note right of Pending
        Initial state.
        Created by EvaluateExportRules
        during Full/Delta Sync.
    end note
    note right of Exported
        Awaiting confirmation.
        Confirming import checks if
        CSO attributes match expected values.
    end note
    note left of ExportNotConfirmed
        Retryable failure.
        Will be re-exported after
        exponential backoff delay.
    end note
    note left of Failed
        Permanent failure.
        Requires manual intervention.
        RPEI: ExportConfirmationFailed
    end note

Full Lifecycle Across Operations¶

A Pending Export's journey typically spans three separate Run Profile executions:

flowchart LR
    subgraph "1. Sync (Full or Delta)"
        SyncStart[MVO attribute changes<br/>during inbound flow<br/>incl. attribute recall +<br/>#91 next-contributor re-election] --> CheckDelete{MVO queued for<br/>immediate deletion?}
        CheckDelete -->|Yes| SkipDelete[Skip export evaluation<br/>MVO about to be deleted;<br/>work would be discarded #390<br/>No PE created]
        CheckDelete -->|No| EvalExport[EvaluateExportRules:<br/>Find export Synchronisation Rules<br/>for MVO type]
        EvalExport --> InScope{MVO in scope<br/>for export rule?}
        InScope -->|No| EvalDeprov[Evaluate deprovisioning:<br/>Create Delete PE if CSO exists]
        InScope -->|Yes| MapAttrs[Map MVO attributes<br/>to CSO attributes<br/>via export Synchronisation Rule mappings]
        MapAttrs --> NetChange{No-net-change<br/>detection}
        NetChange -->|CSO already current| Skip[Skip - no PE created<br/>Target already has correct values]
        NetChange -->|Changes needed| CheckExisting{Existing CSO<br/>in target system?}
        CheckExisting -->|Yes| CreateUpdatePE[Create PE:<br/>ChangeType = Update<br/>Status = Pending]
        CheckExisting -->|No| CreateCreatePE[Create PE:<br/>ChangeType = Create<br/>Status = Pending<br/>Provision new CSO]
        CreateUpdatePE --> FlushReconcile
        CreateCreatePE --> FlushReconcile
        EvalDeprov --> FlushReconcile
        FlushReconcile[Flush-time reconciliation:<br/>CREATE+DELETE for same CSO<br/>cancels both no net change<br/>UPDATE+DELETE cancels UPDATE<br/>keeps DELETE]
        FlushReconcile --> PersistPE[Persist remaining PEs]
    end

    subgraph "2. Export"
        GetExecutable[Get executable PEs:<br/>Status = Pending or<br/>ExportNotConfirmed<br/>NextRetryAt <= now] --> PreExportReconcile[Pre-export reconciliation:<br/>Catch CREATE+DELETE and<br/>UPDATE+DELETE pairs persisted<br/>across different sync runs]
        PreExportReconcile --> MarkExec[Mark batch:<br/>Status = Executing]
        MarkExec --> ConnExport[Connector executes<br/>export operations]
        ConnExport --> Success{Success?}
        Success -->|Yes, Create| ProvResult[Status = Exported<br/>Capture new external ID<br/>RPEI: Exported]
        Success -->|Yes, Update| ExpResult[Status = Exported<br/>RPEI: Exported]
        Success -->|Yes, Delete| DeprovResult[Delete PE + CSO<br/>RPEI: Deprovisioned]
        Success -->|No| FailResult[Increment ErrorCount<br/>Set NextRetryAt<br/>Status = ExportNotConfirmed]
        ProvResult --> OptimisticApply
        ExpResult --> OptimisticApply[Optimistic apply #1079:<br/>Project exported attribute<br/>changes onto the CSO's<br/>in-memory + persisted values<br/>calls-based connectors only;<br/>never stamps LastUpdated/Status]
    end

    subgraph "3. Confirming Import"
        ImportData[Import fresh data<br/>from target system] --> Reconcile[PendingExportReconciliationService:<br/>Compare each PE attribute<br/>against imported CSO values]
        Reconcile --> AllMatch{All attributes<br/>confirmed?}
        AllMatch -->|Yes| DeletePE[Delete PE<br/>Export confirmed<br/>PE lifecycle complete]
        AllMatch -->|Partial| PartialConfirm[Remove confirmed attributes<br/>Keep unconfirmed<br/>Change Create to Update<br/>Status = ExportNotConfirmed]
        AllMatch -->|None| NoneConfirm[Keep all attributes<br/>Increment error count<br/>Status = ExportNotConfirmed]
    end

    PersistPE --> GetExecutable
    OptimisticApply --> ImportData
    FailResult -.->|Next export run<br/>after backoff| GetExecutable
    PartialConfirm -.->|Next export run| GetExecutable
    NoneConfirm -.->|Next export run| GetExecutable

Pending Export Confirmation During Sync¶

During Full/Delta Sync, Pending Exports are also checked for confirmation (separate from the confirming import path above). This uses ISyncEngine.EvaluatePendingExportConfirmation for the pure comparison logic, invoked from SyncTaskProcessorBase:

flowchart TD
    Start([ProcessPendingExport<br/>for each CSO]) --> LookupPE[Lookup Pending Exports<br/>for this CSO from<br/>pre-loaded dictionary]
    LookupPE --> HasPE{PE exists<br/>for CSO?}
    HasPE -->|No| Done([Skip])

    HasPE -->|Yes| CheckStatus{PE<br/>status?}
    CheckStatus -->|Pending| SkipPending[Skip - not yet exported<br/>Nothing to confirm]
    CheckStatus -->|Exported| SkipExported[Skip - awaiting<br/>confirmation via import<br/>reconciliation service]
    CheckStatus -->|ExportNotConfirmed| CompareAttrs[For each attribute change:<br/>Compare expected value<br/>against CSO current value]

    CompareAttrs --> MatchResult{All attributes<br/>confirmed?}
    MatchResult -->|All confirmed| QueueDelete[Queue PE for<br/>batch deletion]
    MatchResult -->|Some confirmed| QueuePartialUpdate[Remove confirmed attributes<br/>If Create, change to Update<br/>Increment error count<br/>Queue for batch update]
    MatchResult -->|None confirmed| QueueFullUpdate[Increment error count<br/>Queue for batch update]

    QueueDelete --> Done
    QueuePartialUpdate --> Done
    QueueFullUpdate --> Done

Attribute-Level Status Tracking¶

Each attribute change within a Pending Export has its own status, enabling partial confirmation:

stateDiagram-v2
    [*] --> Pending: Attribute change created

    Pending --> ExportedPendingConfirmation: Export run executes<br/>successfully

    ExportedPendingConfirmation --> [*]: Confirming import<br/>confirms value matches<br/>(attribute change removed from PE)

    ExportedPendingConfirmation --> ExportedNotConfirmed: Confirming import<br/>finds value mismatch

    ExportedNotConfirmed --> Pending: Reasserted on<br/>next export run

    ExportedNotConfirmed --> Failed: Max retries exceeded

    Failed --> [*]: Manual intervention

Change Types¶

Type	When Created	What Happens
Create	No CSO exists in target system for this MVO	Provisions new object in target system. Connector creates object + sets attributes. PE captures DN template + all attributes.
Update	CSO exists but attributes differ from MVO values	Updates existing object attributes. Only changed attributes are included. No-net-change detection avoids unnecessary exports.
Delete	MVO deletion rule triggered, or MVO falls out of export scope	Removes object from target system. Created by EvaluateMvoDeletionAsync or EvaluateOutOfScopeExportsAsync. PE deleted along with CSO on success.

Drift Detection Creates Corrective Exports¶

During sync, drift detection can also create Pending Exports:

flowchart TD
    DriftCheck[EvaluateDriftAndEnforceState<br/>during sync CSO processing] --> CompareCSO[Compare CSO attribute values<br/>against expected MVO values<br/>using EnforceState export rules]
    CompareCSO --> Drifted{CSO value<br/>differs from<br/>expected?}
    Drifted -->|No| NoDrift([No action])
    Drifted -->|Yes| CheckContributor{Is this system<br/>a legitimate contributor<br/>for this attribute?}
    CheckContributor -->|Yes| LegitChange([Skip - legitimate import<br/>from authoritative source])
    CheckContributor -->|No| CreateCorrective[Create corrective PE:<br/>ChangeType = Update<br/>Status = Pending<br/>RPEI: DriftCorrection]

Drift Correction and Export Evaluation Merge¶

When both drift corrections and export evaluation produce changes for the same Pending Export, they are merged at the value level using composite keys:

flowchart TD
    DriftChanges[Drift corrections<br/>e.g., 117 member removals] --> MergeKey[Merge key =<br/>AttributeId + value identity<br/>e.g., member:user1, member:user2]
    ExportChanges[Export evaluation changes<br/>e.g., 1 member addition] --> MergeKey
    MergeKey --> Deduplicate[Union with value-level dedup<br/>All contributions preserved]
    Deduplicate --> MergedPE[Merged PE contains<br/>all 117 removals + 1 addition]

This prevents silent loss of drift corrections when merging with export evaluation changes. Previously, merging by AttributeId alone would keep only one side's changes for multi-valued attributes.

Key Design Decisions¶

Three-operation lifecycle
A Pending Export typically spans Sync (creation), Export (execution), and Confirming Import (confirmation). This design ensures changes are verified end-to-end.
Partial confirmation
Individual attribute changes can be confirmed independently. If 3 out of 5 attributes match the target system, only the 2 unconfirmed attributes remain on the Pending Export for retry.
Create-to-Update demotion
When a Create PE is partially confirmed (object was created but some attributes didn't take), it's demoted to an Update PE. This prevents the next export from trying to create an already-existing object.
No-net-change detection
Before creating a PE during sync, the system checks if the target CSO already has the expected values (using pre-cached data in ExportEvaluationCache). This avoids unnecessary export operations and reduces connector load.
Drift correction
When EnforceState is enabled on an export Synchronisation Rule and the CSO has values that don't match the MVO, a corrective PE is created to reassert the correct values. This detects and corrects unauthorised changes made directly in target systems.
Attribute recall and re-election
When a Connected System Object disconnects (obsoleted, or fallen out of scope), JIM recalls the values that system contributed. If a lower-priority Synchronisation Rule still contributes, the next contributor is re-elected in the same sync run (Attribute Priority, #91) and export evaluation stages a change-of-value on the target; if no contributor survives, the attribute is cleared and export evaluation stages a null-clear. Export evaluation is no longer skipped for recall. Expression-based mappings (e.g. DN templates) are protected from evaluating against post-recall nulls by re-election supplying a replacement value, by grace-period freezing of identity-critical single-source attributes, and by the #390 skip of recall when the MVO is about to be deleted.
Value-level drift merge
When merging drift corrections with export evaluation changes, the merge key is a composite of AttributeId + value identity (not just AttributeId). This prevents silent loss of multi-valued attribute drift corrections; e.g., 117 member removals would be dropped if merged by AttributeId alone.
Pre-export CREATE→DELETE reconciliation (#218)
Dual-layer reconciliation cancels contradictory Pending Exports before they reach the connector. At flush time (during sync), deferred CREATE/UPDATE PEs are checked against persisted DELETE PEs for the same CSO: CREATE+DELETE pairs cancel both (no net change), UPDATE+DELETE cancels the UPDATE (deletion still needed). At export time, the same logic runs across all Pending Exports to catch pairs persisted in different sync runs. This prevents unnecessary export operations and connector errors.
Exponential backoff
Failed exports use increasing retry delays (NextRetryAt) to avoid hammering a target system that's experiencing issues.
Optimistic export apply (#1079)
On export success, ExportExecutionServer.ProcessBatchSuccessAsync projects the exported attribute changes onto the CSO's attribute values (add-if-absent, replace-on-Update, delete-if-present, delete-all) at export time rather than waiting for the confirming import to re-materialise them. This means the confirming import's "Compare each import attribute against existing CSO values" step (see Full Import Flow) usually finds nothing to do, avoiding millions of redundant attribute value inserts at scale. Two rules keep this safe:
The parent CSO row (Status, LastUpdated, and every other field) is never touched. LastUpdated staying untouched is what lets a no-op confirming import skip the object entirely under the Full Synchronisation unchanged-object watermark; a status transition (e.g. PendingProvisioning → Normal) is still a genuine change and is stamped by the confirming import as before.
Only the calls-based connector paths (sequential, parallel, deferred references) apply; the file-connector export path is excluded because its batch loader does not load the CSO's attribute-value graph, and widening that Include for every file export is a memory-profile trade-off not taken here. File-connector exports keep the pre-#1079 behaviour: the confirming import re-materialises the CSO's attribute values as it always did.

A failure during optimistic apply is caught, logged as a Warning, and never fails the batch, the Pending Export updates, or the Activity - the export itself already succeeded against the Connected System, and the confirming import self-heals by re-materialising the values regardless.

Reference-typed attribute changes (for example a group's member list) resolve their Connected System Object Id purely from PendingExportAttributeValueChange.ResolvedReferenceCsoId, a persisted column stamped once at the point of resolution (ExportExecutionServer.TryResolveReferencesFromLookup, SPEC-1079B). An earlier version of optimistic apply kept this as an in-memory-only hint and fell back to a database lookup whenever a change crossed a persistence boundary without it (cross-run retry, or the parallel batch path's per-batch fresh-context reload); that fallback queried the whole Connected System once per export run, and an even earlier per-batch version of the same fallback cost 10-15 seconds per call over an unindexed scan at Scale500k25kGroups (2026-07-21). Persisting the column removes the loss the fallback was compensating for, so no database lookup is needed here at all: a Pending Export whose Reference change has never been resolved simply applies unresolved (matched on UnresolvedReferenceValue, self-healing on the next import) rather than triggering a lookup. No foreign key or index backs the column - see its XML doc for the trade-off.

Optimistic apply means the confirming import's update path still hydrates every matched CSO's full attribute-value graph to run the diff, even though the diff usually confirms there's nothing to change. At Scale500k25kGroups this held ~9.8M attribute value objects in memory simultaneously across a 524,997-CSO run, tripling the confirming import's live heap versus the pre-#1079 baseline and causing an OOM kill. SyncImportTaskProcessor.PerformImportAsync now releases each update-path CSO's hydrated attribute values once its batch has been persisted (ReleaseHydratedAttributeValues), well after the diff, the change-history write, and the CSO cache refresh have all read what they needed - reconciliation, which runs afterwards, is unaffected because it reloads each CSO's attribute values fresh from the database per page rather than reading the released in-memory copies.

OptimisticExportApplyCalculator.ApplyAdd/ApplyRemove originally called SyncEngine.ValueExistsOnCso/a linear scan once per attribute change, while every accepted Add appended to that same growing per-attribute list - O(M^2) for a Pending Export with M Add changes. The live database has groups with up to 495,008 members, and a full-scale run measured 255 slow OptimisticApply instances totalling 77.5 minutes, all in the group-batch wave. The calculator now builds a per-attribute index once per Pending Export (a Dictionary of value to matching row instances, keyed exactly like ValueExistsOnCso's per-type comparisons - DateTime on Ticks, Reference on the CSO row's UnresolvedReferenceValue, Binary left as a linear SequenceEqual scan since byte arrays do not hash usefully and are never large multi-valued sets) and maintains it incrementally as changes are applied, turning the existence/match check into an O(1) average-case lookup.