# Troubleshooting Oracle RMAN: A Full FRA, Missing Image Copies, and a Backup That Still Succeeded

```text
ORA-19809: limit exceeded for recovery files
ORA-19804: cannot reclaim 4784601600 bytes disk space
           from 300647710720 bytes limit
```

Further investigation uncovered three separate problems: an exhausted Fast Recovery Area quota, a physically full filesystem, and missing image copies that RMAN still considered available.

The most useful lesson from this incident was that **a successful incremental backup does not, by itself, prove that its recovery chain is usable**.

The examples below use simplified paths. Check storage availability and recovery requirements before applying cleanup commands.

**The backup strategy**

The database used incrementally updated image copies with a seven-day recovery window:

```text
RUN {
  RECOVER COPY OF DATABASE
    WITH TAG 'IMAGE_COPY'
    UNTIL TIME 'SYSDATE - 7';

  BACKUP INCREMENTAL LEVEL 1
    FOR RECOVER OF COPY WITH TAG 'IMAGE_COPY'
    DATABASE
    PLUS ARCHIVELOG DELETE ALL INPUT;

  DELETE NOPROMPT OBSOLETE;
}
```

The recovery step applies eligible incremental backups to existing image copies, keeping them behind the current database by the specified interval. The backup step creates new incrementals—or a new baseline image copy where no eligible baseline exists.

This strategy depends on keeping the image copies, incremental backups, and required archived logs available together. [Oracle’s incrementally updated backup documentation](https://docs.oracle.com/en/database/oracle/oracle-database/19/bradv/backing-up-database.html)

**First failure: the FRA quota was exhausted**

We checked the FRA in SQL\*Plus, connected to the CDB root:

```sql
SELECT name,
       ROUND(space_limit / POWER(1024,3), 2) AS limit_gib,
       ROUND(space_used / POWER(1024,3), 2) AS used_gib,
       ROUND(space_reclaimable / POWER(1024,3), 2)
         AS reclaimable_gib
FROM v$recovery_file_dest;

SELECT file_type,
       percent_space_used,
       percent_space_reclaimable,
       number_of_files
FROM v$recovery_area_usage;
```

The results explained the failure:

| Metric | Value |
| --- | --- |
| FRA quota | 280 GiB |
| Used space | 279.45 GiB |
| Reclaimable space | 0 GiB |
| Archived logs | Approximately 225 GiB |
| Image copies | Approximately 36.4 GiB |
| Backup pieces | Approximately 17.4 GiB |

Oracle could not reclaim enough space within its configured quota.

An FRA quota and filesystem capacity are separate limits. Increasing `DB_RECOVERY_FILE_DEST_SIZE` can help when the filesystem has spare capacity; it does not create physical disk space. [Oracle’s ORA-19809 reference](https://docs.oracle.com/en/error-help/db/ora-19809/)

**Shortening retention did not solve it**

The recovery window was temporarily reduced from seven days to five, then three. Each attempt produced:

```text
RMAN> REPORT OBSOLETE;

no obsolete backups found
```

A recovery window is not a rule to delete everything older than a certain date. Older backups and archived logs may still be required to recover to a point inside the window.

We restored the seven-day policy:

```text
CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 7 DAYS;
```

Changing the recovery requirement simply to force cleanup would not repair the underlying backup chain. [Oracle’s backup maintenance guide](https://docs.oracle.com/en/database/oracle/oracle-database/19/bradv/maintaining-rman-backups.html)

**Second failure: a missing image copy**

The copy-recovery step then failed with:

```text
ORA-19870: error while restoring backup piece ...
ORA-19625: error identifying file .../datafile/lob_copy.dbf
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
```

The backup piece named in `ORA-19870` was being processed when the failure occurred. The more specific error identified a missing `.dbf` file.

We checked whether that path belonged to the live database or a backup copy:

```sql
SELECT file#, name
FROM v$datafile
WHERE name = '/fra/PROD/datafile/lob_copy.dbf';

SELECT file#, name, status, tag, checkpoint_time
FROM v$datafile_copy
WHERE name = '/fra/PROD/datafile/lob_copy.dbf';
```

The path was registered as an image copy of datafile 162, tagged `IMAGE_COPY`. Its status was `A`, meaning RMAN considered it available.

The operating system could not find it.

That distinction matters: a repository entry records what RMAN knows about a file. It does not continuously verify that the file remains on disk.

**Repairing the missing baseline**

Before updating repository records, confirm that the expected filesystem is mounted. A temporarily unavailable mount should be restored, not treated as permanently lost storage.

For a genuinely missing copy, we used a targeted crosscheck:

```text
CROSSCHECK DATAFILECOPY '/fra/PROD/datafile/lob_copy.dbf';

LIST COPY OF DATAFILE 162;
```

RMAN changed its status to `X`, meaning expired. We then removed the stale record:

```text
DELETE EXPIRED DATAFILECOPY '/fra/PROD/datafile/lob_copy.dbf';
```

This does not recover disk space occupied by an existing file—the copy was already missing. It reconciles RMAN’s records with storage. [Oracle’s CROSSCHECK reference](https://docs.oracle.com/en/database/oracle/oracle-database/19/rcmrf/CROSSCHECK.html)

Once sufficient space was available, we rebuilt the baseline:

```text
BACKUP INCREMENTAL LEVEL 1
  FOR RECOVER OF COPY WITH TAG 'IMAGE_COPY'
  DATAFILE 162;
```

RMAN reported:

```text
no parent backup or copy of datafile 162 found
channel ORA_DISK_1: starting datafile copy
...
datafile copy complete
Finished backup
```

Although the command specifies level 1, `FOR RECOVER OF COPY` creates a level-0 image copy when no eligible baseline exists. [Oracle’s BACKUP reference](https://docs.oracle.com/en/database/oracle/oracle-database/19/rcmrf/BACKUP.html)

The replacement established a new baseline. It did not restore the historical recovery coverage of the missing copy.

**Third failure: the filesystem was full**

The next backup attempt failed differently:

```text
ORA-19502: write error on file ".../backupset/...bkp"
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 28: No space left on device
```

This time, the filesystem confirmed the problem:

```text
Filesystem                         Size  Used  Avail  Use%  Mounted on
/dev/mapper/VolGroupB0-LogVolB0      295G  281G   1.5G  100%  /b0
```

Increasing the FRA quota alone could not fix this failure.

The practical choices were to expand the filesystem or back up archived logs to separate storage and remove eligible source logs after successful backup.

A template for the second option is:

```text
BACKUP ARCHIVELOG ALL
  FORMAT '/separate_backup_mount/PROD/arch_%U.bkp'
  FILESPERSET 10
  DELETE INPUT;
```

The destination must exist, be writable, and have enough capacity. Any standby or replication requirements must also be accounted for before deleting source logs.

`DELETE INPUT` removes the particular archived-log copies successfully backed up. `DELETE ALL INPUT` has a broader effect: it can remove copies of those logs from all archiving destinations. [Oracle’s archived-log backup documentation](https://docs.oracle.com/en/database/oracle/oracle-database/19/bradv/backing-up-database.html)

The resulting backup pieces should remain on the separate backup storage. Copying them back into the full FRA would consume the space just recovered.

**The database backup succeeded—but recovery still needed attention**

After the archive-backup work, this command completed successfully:

```text
BACKUP INCREMENTAL LEVEL 1
  FOR RECOVER OF COPY WITH TAG 'IMAGE_COPY'
  DATABASE;
```

RMAN created incrementals for files with registered baselines and new image copies for files without them. The control-file and SPFILE autobackup also completed.

However, another recovery log revealed a second missing image copy, this time for datafile 60, a PDB’s `SYSAUX` datafile.

The successful database backup had created an incremental for file 60. It had not repaired the missing baseline because RMAN still believed that baseline was available.

Once more than one missing copy had been identified, checking all datafile copies was appropriate:

```text
CROSSCHECK DATAFILECOPY ALL;

LIST EXPIRED COPY OF DATABASE;
```

Expired entries should be investigated before removal. Missing baselines can then be rebuilt where the live datafiles are accessible and storage capacity permits.

The incident had reached a successful database-backup milestone, but complete recovery validation remained outstanding.

**The background script hid failures from its caller**

The shell wrapper contained:

```bash
set -euo pipefail

nohup rman target / log="${LOGFILE}" <<RMAN >/dev/null 2>&1 &
...
RMAN

disown
exit 0
```

The comment claimed that the script would exit if any command failed. That claim was too broad.

Because RMAN was launched asynchronously and detached, the wrapper could return success before RMAN later failed. In this arrangement, exit status zero meant “the job was launched,” not “the backup completed.”

A more dependable approach is to run RMAN in the foreground inside the worker script and let a scheduler or an outer launcher handle background execution. The worker can then capture RMAN’s exit status and report completion or failure accurately.

**What this incident changed**

The follow-up work was clear:

*   Keep archived-log backups on storage with sufficient capacity.
    
*   Monitor both FRA quota and filesystem free space.
    
*   Check image-copy availability when recovery reports missing files.
    
*   Back up archived logs generated during and after an online database backup.
    
*   Verify the copy-recovery step before restoring automatic cleanup.
    
*   Test recovery, rather than treating successful backup creation as proof of recoverability.
    

The database backup eventually completed. The remaining responsibility was to demonstrate that the preserved copies, incrementals, and archived logs could recover the database to the required point in time.