ORA-27102 out of memory on a system with huge memory

  • A+
Categories:Oracle

Today one customer said the data sync of Oracle Dataguard had issue and our guys checked the alert log then sent below errors to me:

Also had some ORA-00313 errors about failure to open data/redo files.

While he also confirmed the instance was opened and was running.

I asked to check the memory usage, detail trace log and system message log file continually.

The system has 200G physical memory and 16G swap while at that time we only got 3M free swap! so we did get memory issue.

Our guys checked the processes using most of the swap and killed some of them while did not fix it as only very small part of swap was freed.

I got more detailed memory info through /proc/meminfo:

If you read the above output carefully, you would already know the reason.

Then I asked our guys to do below actions:

The SGA was about 120G, and the new content of /proc/meminfo was as below:

So this issue was caused by the HugePage memory.

For some reasons, last time when the database instances was started, it did not make use of HugePage memory as there were about 140G free HugePage memory. After the above actions now we had about 20G HugePage memory. It was correct as the SGA size was about 120G.

 

Comment

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen: