Batch delete

Delete data in bulk from Communications Mining using the CLI, either from a single source or bucket by time range, or across whole datasets with age-based retention.

The CLI provides three ways to delete data in bulk, for example when cleaning up historical data or applying a retention policy.

COMMAND	SCOPE	USE IT FOR
re delete bulk	One source	Deleting comments by time range
re delete bulk-emails	One bucket	Deleting raw emails by time range
re prune	Every source and bucket in the datasets you list	Applying age-based retention to a whole dataset, comments and raw emails together

Warning:

re delete bulk and re delete bulk-emails delete as soon as you run them. There is no confirmation prompt, no dry run, and no backup — take one yourself first. Only re prune backs up what it deletes.

This section assumes that you have already installed and configured the CLI. You need read and delete permissions in every project you are deleting from. Restoring from a backup additionally needs permission to upload comments and emails, and Dataset - Review to restore annotations.

For the options a command takes, run re <command> --help, or see the command reference.

Note:

For all three commands, the time period is based on the timestamp field of the comment or email, rather than the datetime it was uploaded to Communications Mining™.

For re delete bulk and re delete bulk-emails, both ends of the range are optional: give neither and the command covers the whole source or bucket. --from-timestamp is inclusive; --to-timestamp is inclusive for comments, and exclusive for emails.

Backing up data before you delete it

Before deleting or modifying your comments, you may optionally want to back up annotated comments, so as not to accidentally lose the manual work of the model trainers:

re get comments \
  <project_name/source_name> \
  --dataset <project_name/dataset_name> \
  --reviewed-only true \
  --file <output_file_name.jsonl>

If the source was added to multiple datasets, you should run the previously mentioned command for each of those datasets.

That command captures the comments --include-annotated=false keeps, not the ones a deletion removes. To capture those, export the same range you are about to delete: for a source, use re get comments with --from-timestamp and --to-timestamp as described in Batch download, and for a bucket:

re get emails \
  <project_name/bucket_name> \
  --from-timestamp FROM_TIMESTAMP \
  --to-timestamp TO_TIMESTAMP \
  --file <output_file_name.jsonl>

Note:

Attachment content is only exported when you pass --attachments true to re get comments, and only uploaded again when you pass --attachments <directory> to re create comments. Without those, a backup holds attachment metadata but not the files themselves.

Deleting comments from a source

Warning:

Deleting annotations changes model performance.

If the comments you want to delete were added to one or more datasets where they could have been annotated, deleting annotated comments will result in a change of model performance in those datasets going forward. Published models will be unaffected.

Optionally, you can configure the CLI to skip annotated comments.

The following command deletes all comments in a source between FROM_TIMESTAMP and TO_TIMESTAMP , excluding annotated comments. The timestamp should be in RFC 3339 format, e.g. 1970-01-02T03:04:05Z.

re delete bulk \
  --source <project_name/source_name> \
  --include-annotated=false \
  --from-timestamp FROM_TIMESTAMP \
  --to-timestamp TO_TIMESTAMP

If you are sure you want to delete annotated comments, you can set --include-annotated=true.

Note:

Here, --include-annotated=false keeps a comment that is annotated in any dataset containing the source, including datasets you cannot see. re prune uses a narrower rule.

Deleting comments does not remove the raw emails they were parsed from.

Deleting emails from a bucket

To delete the raw emails themselves, target the bucket by time range:

re delete bulk-emails \
  --bucket <project_name/bucket_name> \
  --from-timestamp FROM_TIMESTAMP \
  --to-timestamp TO_TIMESTAMP

This deletes every email in the range, including emails that were parsed into sources other than the one you have been working with. Comments already parsed from those emails are not deleted.

To delete individual emails instead, pass up to 32 ids at a time:

re delete emails \
  --bucket <project_name/bucket_name> \
  <email_id>...

Warning:

Always pass at least one id. Neither re delete emails nor re delete comments checks for an empty list, and a delete request that carries no ids risks removing far more than you intended — so take care with unexpanded placeholders and empty shell variables.

Pruning old data across a dataset

re prune applies age-based retention across one or more datasets in a single run, deleting:

• Comments older than the cutoff, from every source in the datasets you list.
• Emails older than the cutoff, from every bucket those sources read from.

The cutoff is the exact moment the run starts, minus --older-than-days, rather than a calendar day boundary.

Warning:

re prune permanently deletes data. The backup it writes is the only way to undo a run, and it contains personal data. Write backups somewhere secure, keep them safe, and always do a dry run first.

How a prune run works

1. Resolve scope. Every source in the datasets you pass to --datasets is in scope, along with every bucket those sources read from.
2. Check for shared sources. If an in-scope source also belongs to a dataset you did not list, the run aborts and names it. Only datasets you have permission to read can be checked.
3. Confirm. The command summarizes the scope and the caveats that apply to it, and waits for you. --dry-run and -y skip this.
4. Back up. First every reviewed comment in each in-scope dataset, then the comments and emails selected for deletion.
5. Verify, then delete. The run checks every backup file against the record count and checksum in the manifest, and a single mismatch aborts before anything is deleted.

A dry run stops after step 4: it writes a real backup and reports what it would delete, but neither verifies nor deletes.

Running a prune

Start with --dry-run.

re prune \
  --datasets <project_name/dataset_name> \
  --older-than-days 730 \
  --backup-dir <backup_directory> \
  --dry-run

When the counts look right, run the same command without --dry-run.

By default a comment is kept, regardless of age, if it is reviewed in one of the datasets you listed. A comment annotated only in a dataset you cannot access does not count, and is deleted. Pass --include-annotated to delete old comments whether or not they are annotated, bearing in mind the effect on model performance described above.

Pruning a single mailbox

--mailbox restricts what is deleted to the data synced from one mailbox, which is useful when a bucket receives more than one mailbox and only one of them needs pruning. The platform filters emails on the exact mailbox name. Comments are matched, case-insensitively, on their Mailbox ID user property, which email parsing sets only when the source's transform tag records the mailbox name.

Note:

A comment that does not carry a matching Mailbox ID is never matched, so if the comments in scope do not carry that property, a mailbox-scoped run deletes none of them. Check the transform tag with re get sources before relying on --mailbox.

--mailbox does not narrow the annotation backup.

What the backup contains

Each run creates a new folder under --backup-dir, named after the UTC time the run started. re prune never reuses an existing folder.

<backup_directory>/20260807T104500Z/
├── manifest.json
├── annotations/<dataset-id>/<source-id>.jsonl
├── deleted-comments/<source-id>.jsonl
└── deleted-emails/<bucket-id>.jsonl

PATH	CONTENTS
manifest.json	Summary of the run, and the index of every backup file.
annotations/	Every reviewed comment in each in-scope dataset, with its annotations, one file per dataset and source — not only the ones being deleted.
deleted-comments/	The comments selected for deletion, one file per source, in the same format as re get comments, but without their annotations.
deleted-emails/	The emails selected for deletion, one file per bucket, including their raw MIME content.

The manifest records the run's parameters (run_id, cutoff, include_annotated, mailbox, datasets), the size of the deletion set (comment_count, email_count), and for every backup file the resource it covers, its file path relative to the backup folder, its record count and a crc32 checksum.

Note:

comment_count and email_count are the size of the deletion set the run selected. The manifest is written before anything is deleted and is not updated afterwards, so they are not a record of what was successfully deleted.

Restoring from a backup

Restoring is manual, and uses the ordinary re create commands. Backup files are named by id: re get datasets, re get sources and re get buckets list ids alongside names, and the manifest gives the source or bucket each deletion-set file covers in its resource field. Each file's count in the manifest is how many records it holds, which is worth comparing against what you restore.

Restore deleted comments to their source:

re create comments \
  --source <project_name/source_name> \
  --file <backup_directory>/<run_id>/deleted-comments/<source-id>.jsonl

A default prune keeps the comments annotated in the datasets you listed, so annotations only need restoring after a run with --include-annotated. Restore them from the annotation backup — re create annotations reads that file and uploads only the annotations, leaving the comments themselves untouched:

re create annotations \
  --source <project_name/source_name> \
  --dataset <project_name/dataset_name> \
  --file <backup_directory>/<run_id>/annotations/<dataset-id>/<source-id>.jsonl

Restore deleted emails to their bucket:

re create emails \
  --bucket <project_name/bucket_name> \
  --file <backup_directory>/<run_id>/deleted-emails/<bucket-id>.jsonl

Warning:

An annotation backup covers every reviewed comment in the dataset, not only the deleted ones, so restoring one re-applies the annotations as they were at the time of the run. Review work done since the run, on a comment that is in the backup, is overwritten.

Uploading raw emails back into a bucket causes the platform to parse them into the sources that read from that bucket, recreating comments. Do not restore both the emails and the comments for the same source unless you intend to. If you restore the emails rather than the comments, wait for the comments to reappear before restoring annotations, which can only be applied to comments that already exist.

re create comments and re create emails are uploads, so the CLI asks you to consent to the AI unit charges for them. re create annotations does not charge.

Limitations and caveats

Backup and restore:

• Comment attachment content is not backed up. Only attachment metadata is, so attachment content cannot be restored. Emails are backed up whole, so attachments carried in their MIME content are preserved.
• Dismissed extraction field annotations are not restored. They are present in the backup, but the upload format has no way to reapply them.
• A dry run writes a real backup. It reads all the data it would delete and writes it to --backup-dir, so treat its output as securely as any other backup.

Deletion scope and behavior:

• Only annotations in datasets you listed protect a comment. A comment annotated only in a dataset you cannot access is treated as un-annotated: it is deleted, and that annotation is not backed up.
• Email deletion is whole-bucket by age. Without --mailbox, every email dated before the cutoff is deleted from each in-scope bucket, including emails that feed other sources or datasets, whether or not those are in scope.
• Sources outside the listed datasets are out of scope. Their comments are not deleted, even when the emails in their bucket are. This can leave comments whose raw email no longer exists.
• The shared-source check only sees what you can read. If an in-scope source also belongs to a dataset in a project you cannot access, that dataset loses the comments deleted here, and the run cannot warn you.
• Deletion is not transactional. If it fails partway, for example on a network error, the data already deleted stays deleted. The backup is intact, so rerun the command, or restore from the backup.