Troubleshooting Oracle RMAN: A Full FRA, Missing Image Copies, and a Backup That Still Succeeded

ORA-19809: limit exceeded for recovery files
ORA-19804: cannot reclaim 4784601600 bytes disk space
           from 300647710720 bytes limit

Further investigation uncovered three separate problems: an exhausted Fast Recovery Area quota, a physically full filesystem, and missing image copies that RMAN still considered available.

The most useful lesson from this incident was that a successful incremental backup does not, by itself, prove that its recovery chain is usable.

The examples below use simplified paths. Check storage availability and recovery requirements before applying cleanup commands.

The backup strategy

The database used incrementally updated image copies with a seven-day recovery window:

RUN {
  RECOVER COPY OF DATABASE
    WITH TAG 'IMAGE_COPY'
    UNTIL TIME 'SYSDATE - 7';

  BACKUP INCREMENTAL LEVEL 1
    FOR RECOVER OF COPY WITH TAG 'IMAGE_COPY'
    DATABASE
    PLUS ARCHIVELOG DELETE ALL INPUT;

  DELETE NOPROMPT OBSOLETE;
}

The recovery step applies eligible incremental backups to existing image copies, keeping them behind the current database by the specified interval. The backup step creates new incrementals—or a new baseline image copy where no eligible baseline exists.

This strategy depends on keeping the image copies, incremental backups, and required archived logs available together. Oracle’s incrementally updated backup documentation

First failure: the FRA quota was exhausted

We checked the FRA in SQL*Plus, connected to the CDB root:

SELECT name,
       ROUND(space_limit / POWER(1024,3), 2) AS limit_gib,
       ROUND(space_used / POWER(1024,3), 2) AS used_gib,
       ROUND(space_reclaimable / POWER(1024,3), 2)
         AS reclaimable_gib
FROM v$recovery_file_dest;

SELECT file_type,
       percent_space_used,
       percent_space_reclaimable,
       number_of_files
FROM v$recovery_area_usage;

The results explained the failure:

Oracle could not reclaim enough space within its configured quota.

An FRA quota and filesystem capacity are separate limits. Increasing DB_RECOVERY_FILE_DEST_SIZE can help when the filesystem has spare capacity; it does not create physical disk space. Oracle’s ORA-19809 reference

Shortening retention did not solve it

The recovery window was temporarily reduced from seven days to five, then three. Each attempt produced:

RMAN> REPORT OBSOLETE;

no obsolete backups found

A recovery window is not a rule to delete everything older than a certain date. Older backups and archived logs may still be required to recover to a point inside the window.

We restored the seven-day policy:

CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 7 DAYS;

Changing the recovery requirement simply to force cleanup would not repair the underlying backup chain. Oracle’s backup maintenance guide

Second failure: a missing image copy

The copy-recovery step then failed with:

ORA-19870: error while restoring backup piece ...
ORA-19625: error identifying file .../datafile/lob_copy.dbf
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory

The backup piece named in ORA-19870 was being processed when the failure occurred. The more specific error identified a missing .dbf file.

We checked whether that path belonged to the live database or a backup copy:

SELECT file#, name
FROM v$datafile
WHERE name = '/fra/PROD/datafile/lob_copy.dbf';

SELECT file#, name, status, tag, checkpoint_time
FROM v$datafile_copy
WHERE name = '/fra/PROD/datafile/lob_copy.dbf';

The path was registered as an image copy of datafile 162, tagged IMAGE_COPY. Its status was A, meaning RMAN considered it available.

The operating system could not find it.

That distinction matters: a repository entry records what RMAN knows about a file. It does not continuously verify that the file remains on disk.

Repairing the missing baseline

Before updating repository records, confirm that the expected filesystem is mounted. A temporarily unavailable mount should be restored, not treated as permanently lost storage.

For a genuinely missing copy, we used a targeted crosscheck:

CROSSCHECK DATAFILECOPY '/fra/PROD/datafile/lob_copy.dbf';

LIST COPY OF DATAFILE 162;

RMAN changed its status to X, meaning expired. We then removed the stale record:

DELETE EXPIRED DATAFILECOPY '/fra/PROD/datafile/lob_copy.dbf';

This does not recover disk space occupied by an existing file—the copy was already missing. It reconciles RMAN’s records with storage. Oracle’s CROSSCHECK reference

Once sufficient space was available, we rebuilt the baseline:

BACKUP INCREMENTAL LEVEL 1
  FOR RECOVER OF COPY WITH TAG 'IMAGE_COPY'
  DATAFILE 162;

RMAN reported:

no parent backup or copy of datafile 162 found
channel ORA_DISK_1: starting datafile copy
...
datafile copy complete
Finished backup

Although the command specifies level 1, FOR RECOVER OF COPY creates a level-0 image copy when no eligible baseline exists. Oracle’s BACKUP reference

The replacement established a new baseline. It did not restore the historical recovery coverage of the missing copy.

Third failure: the filesystem was full

The next backup attempt failed differently:

ORA-19502: write error on file ".../backupset/...bkp"
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 28: No space left on device

This time, the filesystem confirmed the problem:

Filesystem                         Size  Used  Avail  Use%  Mounted on
/dev/mapper/VolGroupB0-LogVolB0      295G  281G   1.5G  100%  /b0

Increasing the FRA quota alone could not fix this failure.

The practical choices were to expand the filesystem or back up archived logs to separate storage and remove eligible source logs after successful backup.

A template for the second option is:

BACKUP ARCHIVELOG ALL
  FORMAT '/separate_backup_mount/PROD/arch_%U.bkp'
  FILESPERSET 10
  DELETE INPUT;

The destination must exist, be writable, and have enough capacity. Any standby or replication requirements must also be accounted for before deleting source logs.

DELETE INPUT removes the particular archived-log copies successfully backed up. DELETE ALL INPUT has a broader effect: it can remove copies of those logs from all archiving destinations. Oracle’s archived-log backup documentation

The resulting backup pieces should remain on the separate backup storage. Copying them back into the full FRA would consume the space just recovered.

The database backup succeeded—but recovery still needed attention

After the archive-backup work, this command completed successfully:

BACKUP INCREMENTAL LEVEL 1
  FOR RECOVER OF COPY WITH TAG 'IMAGE_COPY'
  DATABASE;

RMAN created incrementals for files with registered baselines and new image copies for files without them. The control-file and SPFILE autobackup also completed.

However, another recovery log revealed a second missing image copy, this time for datafile 60, a PDB’s SYSAUX datafile.

The successful database backup had created an incremental for file 60. It had not repaired the missing baseline because RMAN still believed that baseline was available.

Once more than one missing copy had been identified, checking all datafile copies was appropriate:

CROSSCHECK DATAFILECOPY ALL;

LIST EXPIRED COPY OF DATABASE;

Expired entries should be investigated before removal. Missing baselines can then be rebuilt where the live datafiles are accessible and storage capacity permits.

The incident had reached a successful database-backup milestone, but complete recovery validation remained outstanding.

The background script hid failures from its caller

The shell wrapper contained:

set -euo pipefail

nohup rman target / log="${LOGFILE}" <<RMAN >/dev/null 2>&1 &
...
RMAN

disown
exit 0

The comment claimed that the script would exit if any command failed. That claim was too broad.

Because RMAN was launched asynchronously and detached, the wrapper could return success before RMAN later failed. In this arrangement, exit status zero meant “the job was launched,” not “the backup completed.”

A more dependable approach is to run RMAN in the foreground inside the worker script and let a scheduler or an outer launcher handle background execution. The worker can then capture RMAN’s exit status and report completion or failure accurately.

What this incident changed

The follow-up work was clear:

• Keep archived-log backups on storage with sufficient capacity.

• Monitor both FRA quota and filesystem free space.

• Check image-copy availability when recovery reports missing files.

• Back up archived logs generated during and after an online database backup.

• Verify the copy-recovery step before restoring automatic cleanup.

• Test recovery, rather than treating successful backup creation as proof of recoverability.

The database backup eventually completed. The remaining responsibility was to demonstrate that the preserved copies, incrementals, and archived logs could recover the database to the required point in time.