Skip to content

Migrating Unencrypted Fields

Migrating Unencrypted Fields

Overview

Every previous stage's cardNumber has been transient - CryptoShield requires that of every @Encrypt field, since a non-transient field risks the plaintext getting persisted right alongside its own ciphertext. This stage covers what happens when that requirement collides with reality: some other part of the system (a legacy batch export, a report, a second service) still reads the field directly and isn't ready for it to become transient yet. @EnableMigrationSupport is mango4j-crypto's answer - not a runtime safety mechanism, but a tracked, dated exception with a paper trail. Facilitators should open with the naive failure mode talk/naive-migration/ demonstrates - a table mid-backfill has rows in both states at once - as the broader migration story this annotation is one small, narrowly-scoped piece of, not a replacement for it.

This stage comes as two projects:

  • starter/ - what you work in. It compiles, but throws immediately on startup: both entities have a non-transient @Encrypt field with no @EnableMigrationSupport to excuse it. Look for the // TODO comments.
  • complete/ - the finished reference, where both fields build successfully and log exactly what's expected of them.

Follow along

cd stages/12-Migrating-Unencrypted-Fields/starter
Using an IDE instead? Open stages/12-Migrating-Unencrypted-Fields/starter as its own project.

The rule this stage is about

Every @Encrypt field in every previous stage has been declared transient, without comment - it was just always already that way. AnnotatedEntityManager enforces it: build a CryptoShield over an entity with a non-transient @Encrypt field and nothing else, and it throws immediately, before any encryption ever happens.

InProgressMigrationEntity has a field named cardNumber marked with @Encrypt but it is not transient.
Please mark any fields annotated with @Encrypt as transient

That's what starter/ does, unmodified - this is the exception you're fixing.

The escape hatch

@EnableMigrationSupport(
        completedBy = "2027-01-01",
        justification = "Legacy nightly export job still reads cardNumber directly; not yet updated to call decrypt()",
        ticket = "WORKSHOP-12"
)
@Encrypt
private String cardNumber;

@EnableMigrationSupport doesn't change how cardNumber behaves at runtime - encrypt()/decrypt() read and write it by reflection either way, transient or not. What it changes is registration: instead of throwing, AnnotatedEntityManager logs a message naming the field, the justification, and a ticket reference, and moves on.

Your turn: in starter/.../InProgressMigrationEntity.java, add @EnableMigrationSupport above cardNumber, with a completedBy date in the future.

@EnableMigrationSupport(
        completedBy = "2025-01-01",
        justification = "Same export job, a different field - this one's deadline was missed",
        ticket = "WORKSHOP-13"
)
@Encrypt
private String cardNumber;

Same fix, but with completedBy already in the past - a migration that should have been finished by now, and wasn't.

Your turn: in starter/.../OverdueMigrationEntity.java, add @EnableMigrationSupport above its cardNumber too, with a completedBy date already behind you.

What changes at the deadline

// Building the shield is where @EnableMigrationSupport actually gets
// checked. Watch the console: one WARNING (deadline still ahead) and
// one ERROR (deadline already missed) - neither one stops the build.
CryptoShield cryptoShield = new CryptoShield.Builder()
        .withCryptoKeyProvider(new InMemoryCryptoKeyProvider())
        .withAnnotatedEntities(List.of(InProgressMigrationEntity.class, OverdueMigrationEntity.class))
        .withEncryptionServiceDelegates(List.of(delegate))
        .build();

Building the shield is where both annotations actually get evaluated - watch the console, not the program's System.out lines, since these come from mango4j-crypto's own logger:

WARNING: Field InProgressMigrationEntity.cardNumber is marked with @EnableMigrationSupport. Justification: ... Expected completion: 2027-01-01. Ticket: WORKSHOP-12
SEVERE: Field OverdueMigrationEntity.cardNumber is marked with @EnableMigrationSupport. Justification: ... Expected completion: 2025-01-01. Ticket: WORKSHOP-13 - MIGRATION DEADLINE HAS PASSED!

Before the deadline: a warning. After it: an error - but note what doesn't happen either way. The build still succeeds, and encryption still works:

// Neither field behaves any differently at runtime - transient or
// not, encrypt()/decrypt() read and write it by reflection either way.
InProgressMigrationEntity inProgress = new InProgressMigrationEntity();
inProgress.setCardNumber("5111111111111111");
cryptoShield.encrypt(inProgress);
System.out.println("in-progress field still encrypts fine: " + inProgress.getEncryptedData());

OverdueMigrationEntity overdue = new OverdueMigrationEntity();
overdue.setCardNumber("5222222222222226");
cryptoShield.encrypt(overdue);
System.out.println("overdue field still encrypts fine:     " + overdue.getEncryptedData());

Neither log level blocks anything. @EnableMigrationSupport is a paper trail for whoever's watching application logs or alerting on SEVERE-level messages, not a circuit breaker - the deadline passing is a signal for a person to act on, not a safety mechanism the library enforces on its own.

Finding what still needs backfilling

@EnableMigrationSupport only covers records going through CryptoShield right now. Everything already sitting in the table - rows fetched, until this point, straight off the old plaintext column - still needs encryptedData populated at least once. mango4j-crypto doesn't track backfill progress anywhere, so the application has to answer "which records still need this?" itself, the same way a real completeness query against the table would:

// A handful of "legacy" records: plaintext already sitting in
// cardNumber (as if freshly loaded from the old plaintext column),
// no encryptedData yet.
List<InProgressMigrationEntity> legacyRecords = List.of(
        newLegacyRecord("5111111111111111"),
        newLegacyRecord("5222222222222226"),
        newLegacyRecord("5333333333333335")
);

// Detect which records still need backfilling the same way a real
// query against the table would: encryptedData stays null until
// encrypt() has been called on a record for the first time. There's
// no other signal to go on - mango4j-crypto doesn't track backfill
// progress for you, and this is the *only* reliable one for a
// record that never had any encrypted field before. (A record that
// already has other @Encrypt fields - already CryptoShield-managed,
// encryptedData already non-null - can't be detected this way; that
// case rides along with an ordinary Rekeying: Encryption-style
// sweep instead, since decrypt()/encrypt() already touch it.)
List<InProgressMigrationEntity> stillUnmigrated = legacyRecords.stream()
        .filter(record -> record.getEncryptedData() == null)
        .toList();

stillUnmigrated.forEach(cryptoShield::encrypt);

long remainingUnmigrated = legacyRecords.stream().filter(record -> record.getEncryptedData() == null).count();
System.out.println("legacy records still unmigrated after this sweep: " + remainingUnmigrated + " / " + legacyRecords.size());

encryptedData == null is the signal - it stays null until encrypt() has been called on a record for the first time, and there's no other one to go on for a record that never had any encrypted field before. This matters because it's not the only migration shape: a record that already has other @Encrypt fields already has a non-null encryptedData from those, so this check can't find records newly adding one more field to the mix. That case doesn't need a backfill at all - it rides along with an ordinary Rekeying: Encryption-style sweep instead, since decrypt()/encrypt() already touch every @Encrypt field on the entity together; the only extra step is making sure the new field's legacy value is set on the entity before that encrypt() call, since decrypt() can't produce a value that was never in the original ciphertext.

Unlike that sweep, backfilling a genuinely-new field has nothing to decrypt first - these records only ever had a plaintext value, never any ciphertext, so backfilling one is a single encrypt() call.

Your turn: in starter/.../Main.java, backfill stillUnmigrated: call cryptoShield.encrypt() on each one.

Completing the cutover

package ie.bitstep.mango.workshop;

import ie.bitstep.mango.crypto.annotations.Encrypt;
import ie.bitstep.mango.crypto.annotations.EncryptedData;

/**
 * The other side of the cutover: cardNumber is transient again, with no
 * {@code @EnableMigrationSupport} needed, because nothing outside
 * CryptoShield reads it directly anymore. This is identical in shape to
 * every entity from Real Encryption onward - successfully migrating a
 * field means ending up back at the normal, unremarkable case, not at some
 * new permanent state.
 */
public class MigratedEntity {

    @Encrypt
    private transient String cardNumber;

    @EncryptedData
    private String encryptedData;

    public String getCardNumber() {
        return cardNumber;
    }

    public void setCardNumber(String cardNumber) {
        this.cardNumber = cardNumber;
    }

    public String getEncryptedData() {
        return encryptedData;
    }

    public void setEncryptedData(String encryptedData) {
        this.encryptedData = encryptedData;
    }
}

Once that remaining-unmigrated count is genuinely zero - not just for these three records, every record - the migration is actually done, and @EnableMigrationSupport has served its purpose. MigratedEntity is what cardNumber looks like on the other side of that cutover: transient again, no migration annotation, identical in shape to every entity since Real Encryption. Two things happen together at cutover, in code and in the schema:

  1. The field goes back to plain @Encrypt private transient String cardNumber; - @EnableMigrationSupport comes off entirely, not just its deadline pushed out.
  2. The now-unused plaintext column gets dropped from the database. Nothing in this workshop's plain Java entities has an actual schema to alter, but the code change is the signal that it's safe to: once nothing maps cardNumber to a persisted column anymore, nothing is reading that column either, and keeping a dropped field's data around is pure liability with no upside.
// Only cut over once that count is genuinely zero - not just these
// three records, every record. The field goes back to being fully
// transient, @EnableMigrationSupport comes off, and (in a system
// backed by a real database) the now-unused plaintext column gets
// dropped from the schema. MigratedEntity is what cardNumber looks
// like on the other side of that cutover - identical to every
// entity from Real Encryption onward.
if (remainingUnmigrated > 0) {
    System.out.println("cutover skipped: " + remainingUnmigrated + " record(s) still not backfilled");
} else {
    CryptoShield cutoverShield = new CryptoShield.Builder()
            .withCryptoKeyProvider(new InMemoryCryptoKeyProvider())
            .withAnnotatedEntities(List.of(MigratedEntity.class))
            .withEncryptionServiceDelegates(List.of(delegate))
            .build();

    MigratedEntity migrated = new MigratedEntity();
    migrated.setCardNumber("5444444444444447");
    cutoverShield.encrypt(migrated);
    System.out.println("post-cutover field still encrypts fine: " + migrated.getEncryptedData());
}

Cutover is gated on that count, not run unconditionally - a real migration doesn't get to assume the backfill actually finished just because the sweep ran once.

Running it

starter/ throws before any System.out line is ever reached - the exception shown above, on the first entity CryptoShield tries to register.

After all three changes:

in-progress field still encrypts fine: {"cryptoKeyId":"workshop-encryption-key",...}
overdue field still encrypts fine:     {"cryptoKeyId":"workshop-encryption-key",...}
legacy records still unmigrated after this sweep: 0 / 3
post-cutover field still encrypts fine: {"cryptoKeyId":"workshop-encryption-key",...}

Plus the two log lines from earlier, printed during CryptoShield.Builder().build() before any of those. The last two lines are this addition: the remaining-unmigrated count confirms every legacy record picked up its ciphertext, and MigratedEntity - built with a completely separate CryptoShield that's never even heard of @EnableMigrationSupport - proves the field works exactly like any other stage's once the migration is behind it.

The rest of the migration story

This annotation buys a legacy code path time - it doesn't do the migration itself. Actually moving a genuinely unencrypted field to encrypted, field by field, across records already in production, is what talk/naive-migration/ walks through: a table mid-backfill has rows in both states at once, and a naive load() that assumes everything's already ciphertext throws on the rows the backfill hasn't reached yet. A real migration needs all three pieces demonstrated on this page - a tracked, dated relaxation for whatever still needs direct access, a backfill sweep that reaches every record, and a cutover once it has - plus the piece only talk/naive-migration/ covers: tolerating a table in a mixed state for as long as the backfill is still in progress.

This is the last stage in the hands-on arc. Together with Key Rotation, Rekeying: Encryption, and Rekeying: HMACs, it completes the promise made all the way back in Introduction: pluggable encryption providers, multiple HMAC strategies, rekeying support, and migration of existing unencrypted fields, each demonstrated with real, runnable code rather than just described.