S3 Storage cleanup

The S3 Storage used by the TRON.ASOC system accumulates scan data, such as raw scan results, Reports, SBOM files, custom field files, and other objects. To prevent the amount of stored data from growing uncontrollably, the system uses the following mechanisms to delete data from S3 Storage.

S3 Storage cleanup mechanisms

Storage cleanup uses two independent mechanisms:

• automatic deletion based on time to live (TTL)—this mechanism applies to generated Reports and raw scan results;
• cascading deletion when entities are deleted—raw scan results are deleted when the security check, layer, or project associated with the data is deleted.

S3 physically deletes data asynchronously through an internal task queue. This approach doesn't affect the processing speed of user requests and handles temporary Storage unavailability correctly.

Files to delete

The system deletes the files shown in the following table.

File type	S3 bucket	Object name	TTL deletion	Manual deletion
Raw scan results	scan-results	<scan_result_id>.json	Yes	When the check, layer, or project is deleted
ASOC report	REPORT_BUCKET_NAME (defaults to reports)	Values from report.result.files	Yes	When the report is deleted through the Reports API
SBOM	sbom	<sbom_id>.json	No	When the SBOM is replaced
Custom field file	custom-field-files	storage_path from the database	No	Through FileFieldService

Automatic TTL cleanup doesn't affect SBOMs or custom field files. You can delete them only manually.

Configuring automatic cleanup (TTL)

The core Service's STORAGE_CLEANUP_TTL_DAYS environment variable controls automatic Storage cleanup.

The variable can have the following values:

• 0—automatic TTL cleanup is disabled, but manual deletion continues to work when entities are deleted. This is the default value;
• N > 0—files older than N days are deleted;
• a negative value—automatic TTL cleanup is disabled.

note

Changes to STORAGE_CLEANUP_TTL_DAYS take effect only after you restart the core Service.

tip

Before you enable TTL for a large amount of data, we recommend estimating the number of deletion candidates (dry run) to prevent unexpected load on the worker and S3.

Deployment manifests define the STORAGE_CLEANUP_TTL_DAYS value. You don't need to edit configuration files inside the container.

To configure automatic cleanup for TRON.ASOC installed with Docker Compose, follow these steps:

1. Edit the environment block for the core Service in the docker-compose.yaml file:

services:
  core:
    environment:
      - STORAGE_CLEANUP_TTL_DAYS=${STORAGE_CLEANUP_TTL_DAYS:-0}

2. Apply the new variable value: docker compose up -d core
3. Verify that the variable value was applied: docker compose exec core printenv STORAGE_CLEANUP_TTL_DAYS

To configure automatic cleanup for TRON.ASOC installed with a Helm chart, follow these steps:

1. Edit the values.yaml file:

component:
  asoc-back:
    container:
      asoc-back:
        env:
          STORAGE_CLEANUP_TTL_DAYS: "30"

2. Apply the new variable value: helm upgrade asoc ./asoc -n <namespace> -f values.yaml
3. Verify that the variable value was applied: kubectl exec -n <namespace> deploy/asoc-back -- printenv STORAGE_CLEANUP_TTL_DAYS

warning

This method works with chart version 1.7 and later. Earlier chart versions don't include values in the .env block.

How it works

Task queue

Neither automatic nor manual deletion occurs directly. Instead, the system adds a task containing the bucket, object path, and source type to the file_cleanup_task table. A background worker periodically retrieves pending tasks and physically deletes the data from S3.

The worker starts when the Core Service starts and performs cleanup immediately. It then repeats the cycle every 24 hours until the Core Service stops. If STORAGE_CLEANUP_TTL_DAYS is ≤ 0, the system stops creating new TTL tasks, but it continues to process tasks already in the queue.

Selecting data by TTL

The TTL mechanism doesn't delete data directly. It only creates tasks for the data in the queue:

• raw scan results—the mechanism selects data with the following properties:

  • the created_at field (creation date) is older than the TTL;
  • the status is done (completed), failed, or canceled;
  • the raw_result_deleted_at marker (when to delete the result) isn't set;
  note

The mechanism doesn't include records with an existing cleanup task in the selection again. :::

• Reports—the mechanism selects data with the following properties:

  • the status is done (completed);
  • the updated_at value (update date) is older than the TTL;
  • the files_deleted_at marker (when to delete the files) isn't set.

The system uses the updated_at value to calculate the TTL. After it regenerates a report, the new file receives the full TTL again.

Cascading deletion when entities are deleted

When you delete a security check, layer, or project through the API, deletion and task creation occur separately. PostgreSQL deletes the entity in a transaction, and the system creates tasks to delete the related scan results only after the transaction succeeds. The system returns the HTTP response to the client immediately without waiting for the worker to physically delete the files from S3.

The system deletes records in the same way when you delete a security tool or scan source. It collects the related security checks inside a PostgreSQL transaction and creates tasks only after the transaction completes.

Error handling

Situation	Behavior
The object no longer exists in S3	The system considers the deletion successful and sets the marker
S3 is temporarily unavailable	The task remains in pending status and is logged as an error. The worker retries it during the next pass
The object was deleted, but the PostgreSQL record wasn't updated	Because S3 and PostgreSQL don't support a shared transaction, the object may be deleted while the marker isn't set yet. During the next pass, the worker safely retries the object deletion (idempotently) and completes the task

Monitoring and troubleshooting

Checking the status with SQL

To check the status of data scheduled for deletion, run the following queries in the Core Service database:

-- Состояние результатов сканирования
SELECT id, status, created_at, deleted_at, raw_result_deleted_at
FROM scan_result
ORDER BY created_at DESC;

-- Состояние отчетов
SELECT id, status, updated_at, result, files_deleted_at
FROM report
ORDER BY updated_at DESC;

-- Очередь задач очистки
SELECT id, bucket_name, object_path, source_kind, source_id, source_updated_at, status, deleted_at
FROM file_cleanup_task
ORDER BY created_at DESC;

Known limitations

The following limitations of the current cleanup implementation may affect the speed or completeness of Storage cleanup in certain situations:

1. Risk of losing a task if the Core Service stops unexpectedly. The system adds a file deletion task to the queue after the related operation, such as deleting a project, is committed to the database. If the Core Service stops unexpectedly between the database commit and task creation, the system doesn't create the task. As a result, the files remain in S3, and the system doesn't retain information that it needs to delete them.
2. No batch deletion. The worker deletes objects one at a time instead of in batches. Therefore, deleting a large project with many related files creates a large number of individual tasks, which can significantly increase the time required for complete cleanup.
3. TTL doesn't process historical records deleted earlier. The query that selects artifacts by TTL includes the deleted_at IS NULL condition. As a result, records marked as deleted (soft delete) before the cleanup mechanism was introduced don't meet this condition and aren't processed automatically. You must delete these files separately.
4. Risk when regenerating a report with the same file name. If you regenerate a report with the same file name, a task added to the queue before regeneration could theoretically run after regeneration and delete the new file. This can happen if the task runs before the updated_at protection takes effect. The protection is intended to prevent this situation. 5 No fast retries with increasing backoff. If deletion fails, the task remains in pending status and isn't retried immediately. The next attempt occurs only during the next full worker cycle—that is, approximately 24 hours later.