Migration execution and Engine API

This page explains where and how migration scripts run. When upgrading the Platform, start with the release migration catalog. When changing your own process, start with the procedure for migrating existing cases.

If you only need to run a ready-made script, the Before execution, Steps in Admin, and Execution options sections are sufficient. The chapters on MigrationHelper and Engine configuration are intended for migration script authors.

A backup is mandatory

Before every production migration, create a restorable and verified backup of MongoDB and storage. If the migration affects search, also prepare an Elasticsearch snapshot or a verified full-reindex procedure. Create the backup before the first write and test the restore process in a non-production environment beforehand.

Which mechanism to use

Use the first mechanism available in your deployment:

1. Admin - recommended

Use Admin → Cluster → Migrations for a one-time Platform, release, or operational migration in Enterprise. The run has a status and logs, and the script can be canceled cooperatively. The migration design must still account for partial writes.

2. Versioned startup class

Use class Migration_1_2_0 extends MigrationOrderedCommandLineRunner in a custom Engine build, module, or Community deployment without Admin. A record is created only after success. A failed run may start again on the next startup, so it must be safely repeatable.

3. Process event - last resort

Use Petriflow processEvents / UPLOAD post only for a small migration tightly coupled to the import of one process. It does not record a release migration and runs again on the next import. After a partial failure, it needs its own guard, counters, and safe rerun behavior.

ProcessApplicationStartupRunner is intended for controlled initialization during startup, but it runs on every startup and needs its own persisted guard. A NAE module is suitable for a reusable migration tool with additional dependencies; deploying or updating it requires a worker restart.

The old general recommendation to “run migrations through processEvents” no longer applies to Platform release migrations. In an Enterprise deployment, run published one-time migrations through Admin. A Spring class from an old migration project can still be a useful source of transformation logic, but do not paste the entire class into Admin.

How to read older migration articles

Older materials may contain useful transformation logic, but they are not automatically instructions for the current Enterprise runtime. Before using them, correct these claims in particular:

Older claim or example	Current conclusion
“processEvents are the recommended migration method.”	Use Admin for one-time Enterprise migrations. Reserve processEvents for small, process-owned, idempotent logic.
ApplicationContextProvider.getBean("migrationHelper") in a Petriflow action	Bean lookup may technically work, but it is not part of the stable Action API contract. Admin provides migrationHelper directly.
getNewestVersionByIdentifier(...)	It does not exist in the current API. The replacement is getDefaultVersionByIdentifier(...), but a production release script should still target an exact PetriNet ObjectId; the default may change after another import.
workflowService.save(useCase) in an update* closure	Do not call it. The helper automatically adds the changed document to the bulk write; a manual save is redundant and may complicate concurrent writes.
taskService.reloadTasks(useCase)	The current service signature is reloadTasks(useCase, boolean lazyCaseSave). For migrations, prefer migrationHelper.reloadTasks(useCase, targetNet).
updateNetIgnoreRoles(identifier, "petriNets/process.xml")	fileName is already relative to the petriNets/ classpath directory. Pass only "process.xml".
“Before reloadTasks, the case must already point to the target net.”	False. The helper sets the runtime PetriNet itself. Change the persisted petriNetObjectId separately with migratePetriNet.
caseRef, taskRef, and multichoice are always a List	False. References are List<String>, but multichoice uses Set<I18nString> and multichoice_map uses Set<String>.
“FileFieldValue has no no-argument constructor.”	It does in the current Engine. However, use null for a new empty FILE; an empty FILELIST can be new FileListFieldValue(). Store actual content and metadata through dataService.
A production file created with MockMultipartFile	MockMultipartFile is a testing helper. At runtime, use MultipartFileImpl as shown in the file recipe.
Compound MongoDB indexes are recommended in general	Add an index only for a specific filter and data volume, and after verifying the query plan.

Running a migration in Admin

Before execution

1. For a Platform upgrade, select the exact path in the catalog and proceed only with a migration marked ready.
2. For a process migration, determine the exact source and target PetriNet ObjectId and the expected number of cases.
3. Select the worker that can access the data being migrated.
4. Stop or limit concurrent writes to the affected data.
5. Create a restorable backup of MongoDB and storage and, if required by the recovery plan, an Elasticsearch snapshot.
6. Test the script on an up-to-date copy of production and prepare verification queries.

Steps in Admin

1. Open Cluster → Migrations and the Scripts tab.
2. Select New Migration, name the script, and paste the Groovy code into Executable.
3. Open the migration and select Execute.
4. Add only the expected inputs. A Variable is a reusable cluster value, an Argument applies to one run, and a FILE is bound as java.io.File on the worker.
5. Select the owning worker, configure logging and an optional MongoDB transaction, and start the run.
6. Check the status, exit code, logs, counts, and domain-specific verification step.

Only one migration script can run on a worker at a time. Another run is rejected while the worker is busy; it is not queued.

Execution options

Option	Meaning
Debug mode	Enables breakpoint("name"); Admin allows execution to continue after inspection.
Run in MongoDB transaction	Uses a transaction only with a supported MongoDB replica set. It does not cover storage, Elasticsearch, or other workers.
Return all log entries	Sends all retained migration logs to Admin after execution.
Log batch size	The number of log entries in one transferred batch.
Max log entries in Node	The maximum number of migration log entries retained in worker memory.
Additive Logger	Also sends migration logs to the worker's regular appenders.
Save Output as New Variable	Stores the return value as a new reusable Variable.

DONE means that the script runner did not receive an uncaught error. It does not mean that the script selected the correct documents or performed the correct business transformation.

Admin script format

The Executable field contains the Groovy script body directly:

def count = 0
def query = new org.springframework.data.mongodb.core.query.Query()

mongoTemplate.stream(
    query,
    com.netgrif.application.engine.objects.workflow.domain.Case.class
).withCloseable { cases ->
    cases.each { useCase ->
        cancellable()
        count++
        // transformation
    }
}

log.info('Processed cases: {}', count)
return count

Do not use a @Component wrapper, @Autowired, a migrate() override, or MigrationOrderedCommandLineRunner. Admin directly provides the services described in the next section. Admin wraps the field content in a closure, so do not add import statements. Worker configuration supplies common Engine types; for any other type, use its fully qualified name, as in the example above.

What ScriptDelegate provides

An Admin script directly provides, in particular:

• migrationHelper, petriNetService, workflowService, taskService, dataService;
• mongoTemplate, userService, realmService, processRoleService;
• Elasticsearch case/task services and mapping services;
• log, binding, cancellable(), and breakpoint(name).

Direct services are local to the worker on which the script runs.

The release migration catalog provides the specific user-facing task list for available versions. Successful script compilation alone is not enough to mark a release migration as production-ready.

ClusterMigrationHelper

The Platform worker uses ClusterMigrationHelper, which extends the Engine's MigrationHelper. Currently, only the following operations have cluster-aware delegation:

• updateCasesCursor by processIdentifier or PetriNet ObjectId;
• updateTasksCursor and updateSpecificTasksCursor;
• updateNetIgnoreRoles.

Other helper methods and direct services operate on the selected worker. Therefore, even a cluster-aware cursor does not replace correct owning-worker selection, especially if the script loads targetNet locally before the cursor.

Cursor updates

update*Cursor loads documents using a Mongo cursor, invokes the closure, and saves the changed document with an unordered bulk replaceOne. Cursor overloads have their own default pageSize = 100; configured page sizes apply to predicate-based updateCases and updateTasks. Therefore, pass a different page size to the cursor method explicitly.

Limitations

• The closure modifies the loaded object; its return value is ignored.
• Do not call workflowService.save(useCase) or repository save inside an update* closure; the helper saves the document in a bulk write.
• The bulk write is unordered, so a batch may be partially successful.
• The case bulk operation uses upsert; the task bulk operation does not.
• A direct bulk write bypasses domain actions, events, and the service lifecycle.
• An individual document error either continues or stops the run according to the selected MigrationErrorPolicy; do not rely on the default CONTINUE mode.
• Do not use the cursor helper to change a MongoDB _id.
• Do not call removeCase(useCase) inside the case update* closure. The helper performs an upsert at the end of the page and may recreate the deleted case in MongoDB while it remains deleted from Elasticsearch. Delete it in a separate phase with no subsequent case bulk write.
• A custom pageProcessed closure in low-level iterateCases/iterateTasks takes over bulk-operation processing.
• A concurrent service may save another instance of the same case; separate storage and similar phases for this reason.

Most used methods

Ready-to-adapt snippets for the most frequent operations. Each one belongs inside an Admin script; for the surrounding skeleton, see the procedure for migrating existing cases, and for the per-change logic, see the recipes by change type.

Load the target version once, before the cursor, and bind it to an exact ObjectId. Do not resolve “newest” at runtime - the default version can change with the next import.

def targetNet = petriNetService.get(
    new org.bson.types.ObjectId("TARGET_PETRINET_OBJECT_ID")
)

Updating an existing PetriNet model. fileName is relative to the petriNets/ classpath directory, so do not include that prefix:

migrationHelper.updateNetIgnoreRoles("example_process", "example_process.xml")

Updating the existing cases of one process in MongoDB:

migrationHelper.updateCasesCursor({ useCase ->
    cancellable()
    useCase.title = "${useCase.title} - migrated"
}, "example_process", 100)

Updating all existing tasks of one process in MongoDB:

migrationHelper.updateTasksCursor({ task ->
    cancellable()
    task.title = "${task.title} - migrated"
}, "example_process", 100)

Updating only the tasks of selected transitions:

migrationHelper.updateSpecificTasksCursor({ task ->
    cancellable()
    task.title = "${task.title} - migrated"
}, "example_process", ["t1", "t4"])

Reindexing cases and tasks. Run this as a separate phase, only after the MongoDB phase has succeeded:

migrationHelper.updateCasesCursor({ useCase ->
    migrationHelper.elasticIndex(useCase)
}, "example_process", 100)

migrationHelper.updateTasksCursor({ task ->
    migrationHelper.elasticTaskIndex(task)
}, "example_process", 100)

Reloading tasks, for example after adding a new transition to the process:

migrationHelper.updateCasesCursor({ useCase ->
    migrationHelper.reloadTasks(useCase, targetNet)
}, "example_process", 100)

Migrating existing cases to a newer process version:

migrationHelper.updateCasesCursor({ useCase ->
    migrationHelper.migratePetriNet(useCase, targetNet)
}, "example_process", 100)

Updating the components on DataFields and dataRefs, which is required when the components change in the PetriNet model:

migrationHelper.updateCasesCursor({ useCase ->
    migrationHelper.updateCaseComponents(useCase, targetNet)
}, "example_process", 100)

Do not call a manual save

Inside an update* closure, do not call workflowService.save(useCase) or taskService.reloadTasks(useCase). The helper writes the modified document in a bulk operation, and migrationHelper.reloadTasks(useCase, targetNet) sets the runtime PetriNet itself. The exact effect and the limits of each method are listed below.

Most important helper methods

Cases and tasks

Method	Actual effect
updateCases(update, Predicate)	Selects cases with a QueryDSL predicate and uses the configured cases page size.
updateCasesCursor(update, processIdentifier, pageSize)	Selects cases with the identifier across versions. Cluster-aware in the Platform.
updateCasesCursor(update, petriNetObjectId, pageSize)	Selects cases from one exact version. Cluster-aware in the Platform.
updateTasks(update, Predicate)	Selects tasks with a QueryDSL predicate and uses the configured tasks page size.
updateTasksCursor(update, processIdentifier, pageSize)	Selects tasks whose processId belongs to the current default version.
updateSpecificTasksCursor(...)	The same, restricted to a transition ID.
migratePetriNet(useCase, targetNet)	Changes only useCase.petriNetObjectId.
setPetriNet(useCase, targetNet)	Sets only the runtime PetriNet and initializes its tokens/arcs; it does not change petriNetObjectId.
reloadTasks(useCase, targetNet)	Internally sets the runtime PetriNet and recalculates tasks. It does not have to change the persisted version reference.
updateCasePermissionsFromNet(...)	Replaces case permissions, derives view roles, and sets enabled roles; it does not change user assignments.
updateTasksPermissions(...)	Sets the target processId, roles, and negative view roles on selected existing tasks.
elasticIndex(useCase) / elasticTaskIndex(task)	Passes the document to an Elasticsearch service outside the MongoDB transaction; the current implementation uses a queue.

reloadTasks creates missing executable tasks and removes a disabled task only if it has no assigned user. It does not overwrite an existing task that remains executable. It may trigger an auto-trigger.

Case data and task forms

Method	Actual effect
addTextDataFields(useCase, values)	Adds basic DataField values without model metadata; suitable only for simple text.
deleteDataFields(useCase, ids)	Removes runtime fields from the dataset map.
changeDataFieldsValueFromNumberToText(...)	Converts a non-empty value through double to String.
changeDataFieldsValueFromTextToNumber(...)	Converts to Double; sets an invalid value to null and passes it to the error policy.
changeDataFieldsValueFromEnumerationToMultichoice(...)	Wraps the value in a Set<I18nString>.
changeFileFieldToFileList(...)	Wraps a FILE value in a FILELIST value without moving storage content.
addChoices(...)	Adds I18nString choices to the runtime field.
removeChoices(...)	Removes entries from the runtime value/choices; verify the specific type and historical-data equality before use.
updateCaseComponents(useCase, net)	Sets only components that exist in the target model; it does not clear removed components.
createComponentsMap(net)	Returns only non-null field components from the target model.
createDataRefComponentsMap(net)	Returns only non-null dataRef components by field and transition ID.

A runtime DataField does not store its Petriflow type. The type is read from the PetriNet model; the runtime object stores the value, behavior, choices/options, allowedNets, validations, and components.

Use a FILE value of null for “no file.” An empty FILELIST is represented by new FileListFieldValue(). Although the current FileFieldValue has a no-argument constructor, an empty object with no storage path is not a substitute for saving a file through dataService.

PetriNet and roles

Method	Actual effect
updateNetIgnoreRoles(identifier, fileName, customUpdates)	Updates the default model in place from classpath petriNets/<fileName> and preserves old roles. fileName must no longer contain the petriNets/ prefix.
updateNetIgnoreRoles(identifier, Resource, customUpdates)	First creates a new MAJOR version through a regular import and then merges the default model; this is not a simple in-place import.
updateDataSet(identifier, fileName, customUpdate)	Replaces the default version's dataset from a classpath model.
createRoleInNet(...)	Creates a role and adds it to the default PetriNet; it does not assign the role to users.
createGlobalRole(...)	Creates a global role and adds the global_ prefix if necessary.
updateTransitionRoles(...)	Sets one role's permissions on a transition in the supplied PetriNet.
updateRoleEvents(existing, reimported)	Replaces events of existing roles by role importId.

updateNetIgnoreRoles deliberately ignores new process roles and new actor role references. For a v1 → v2 release migration, it is usually safer to import v2 through the standard deployment flow and migrate existing cases to the exact target model.

Error policy

The default policy is CONTINUE, so an individual error may not stop the run. For a release migration, use an explicit policy and evaluate the errors at the end:

migrationHelper.clearErrors()
migrationHelper.withErrorPolicy(
    com.netgrif.application.engine.migration.model.MigrationErrorPolicy.continueOnError()
) {
    // cursor operations
}

if (migrationHelper.hasErrors()) {
    def errors = migrationHelper.popErrors()
    errors.each { error -> log.error("Migration error: {}", error) }
    throw new IllegalStateException("Migration failed: ${errors.size()} errors")
}

Available modes are THROW_IMMEDIATELY, CONTINUE, THROW_AFTER_LIMIT, and THROW_AFTER_PROCESSING. The example uses an explicit CONTINUE because it evaluates errors immediately after the block. With THROW_AFTER_PROCESSING, the helper throws a MigrationErrorException containing the collected errors at the end of the operation; the following if does not run when an error occurs. Add cancellable() to every long-running loop; cancellation from Admin is cooperative.

A custom try/catch inside an update closure that only logs the error without rethrowing it hides the error from MigrationErrorPolicy. Either use the helper policy or, in catch, add context and rethrow the error. Do not combine the approaches in a way that allows the script to report success despite skipping a document.

The error cache is not cleared automatically, and THROW_AFTER_PROCESSING is evaluated at the end of each helper iteration, not only at the end of the entire script. withErrorPolicy uses ThreadLocal; with custom parallelization, the policy is not propagated to new threads.

Engine helper configuration

These Spring properties are relevant to a custom Engine build or module. In an Admin script, pass critical values, such as the cursor page size, explicitly.

Property under netgrif.engine.migration	Default	Actual use
cases.page-size	100	updateCases and iterateCases; not cursor overloads with their own default.
tasks.page-size	100	updateTasks and iterateTasks; not cursor overloads with their own default.
petri-nets.page-size	100	Predicate-based PetriNet iteration.
error-policy.mode	CONTINUE	Default helper policy.
error-policy.max-errors	0	Limit for THROW_AFTER_LIMIT; zero is evaluated after processing.
error-policy.cache-errors	true	Determines whether the helper stores the error in the migration error cache.
error-policy.throw-original	false	Rethrows the original exception only when it is a RuntimeException.

skip, evict-caches, and shutdown-after-migration in MigrationProperties are not helper behavior by themselves. An application runner must process them explicitly; the current MigrationOrderedCommandLineRunner does not use them as general execution switches.

Performance and indexes

To improve migration performance, we recommend creating a compound MongoDB index on processIdentifier and _id:

db.case.createIndex(
    { "processIdentifier": 1, "_id": 1 },
    { background: true }
)

The background: true option creates the index in the background, so the database does not need to be taken offline while the index is being built.

For large migrations, start with pageSize = 100, log in batches, do not wrap millions of writes in one long-running transaction, and separate the MongoDB, storage, and Elasticsearch phases.

Migrations in a custom build or module

If Admin is unavailable, the second recommended option is a versioned startup class such as Migration_1_2_0. It can extend MigrationOrderedCommandLineRunner:

@Component
class MigrateProcessA extends MigrationOrderedCommandLineRunner {
    @Autowired MigrationHelper migrationHelper

    @Override
    void migrate() {
        migrationHelper.updateCasesCursor({ useCase ->
            // idempotent change
        }, "process_a", 100)
    }
}

After success, the class's simple name is stored. Renaming the class is treated as a new migration. If migrate() throws an error, no record is created and the migration may run again on the next startup. Therefore, this runner does not replace an idempotent script, a safe resume procedure, or a backup. ProcessApplicationStartupRunner does not provide this record and needs its own guard and an explicitly resolved runner order.