Magento 2.4 critical load of RAM triggering OOM killer due to CSP module 100.4.0

[onepack]

Preconditions (*)

1. Magento 2.4.0
2. PHP 7.3 and 7.4
3. Redis enabled for FPC and backend cache
4. Elasticsearch 7.8.1
5. Database mariadb 10.4
6. 8gb to 10gb of VPS ram with Magento having 2gb assigned.

Steps to reproduce (*)

1. Set the shop to production mode
2. Have around 2000+ products in multiple categories
3. Clear all caches and also redis-cli flushall
4. Open a dropdown with categories and click them open in a new tab one by one.
5. Search a couple of times and wait for the result.

Expected result (*)

1. The expected result is that all 15 categories would open one by one and look ok
2. The search result would look ok.

Actual result (*)

1. The actual result is that maybe 12 categories would open ok but around 2 or 3 would look completely broken with broken styling or even a 500 server error.
2. The search result would look broken.
3. Sometimes the OOM killer was triggered that would sacrifice the Elasticsearch etc.

---

Please provide Severity assessment for the Issue as Reporter. This information will help during Confirmation and Issue triage processes.

• Severity: S0 - Affects critical data or functionality and leaves users without workaround.
• [X ] Severity: S1 - Affects critical data or functionality and forces users to employ a workaround.
• Severity: S2 - Affects non-critical data or functionality and forces users to employ a workaround.
• Severity: S3 - Affects non-critical data or functionality and does not force users to employ a workaround.
• Severity: S4 - Affects aesthetics, professional look and feel, “quality” or “usability”.

We found out via SSH top that the RAM consumption was peaking during the category generation on the frontend (from uncachd to cached).
First we thought it was a Redis issue because 9 out of ten times the page would look broken like we have seen in the M1 days because of a Redis issue.
But actually "the broken page layout" was the result of PHP running out of memory.
The result would get cached and stay in the cache until the redis cache was cleared.

After a lot of digging I found the source of the issue in:
vendor/magento/module-csp/Model/BlockCache.php
Under the function: load($identifier)
This function uses a crazy amound of RAM.
Quick workaround is disabling the module to keep the shop RAM usage healthy.

The log errors show that this function is causing issues.

I copied the log errors from another user running into the same issue:

[Mon Aug 17 19:40:50.052935 2020] [php7:error] [pid 16932:tid 140669993522944] [client 127.0.0.1:40654] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 48238592 bytes) in /var/www/html/vendor/magento/framework/Serialize/Serializer/Json.php on line 24
[Mon Aug 17 19:40:51.592030 2020] [php7:error] [pid 16933:tid 140670169769728] [client 127.0.0.1:40666] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/vendor/magento/module-csp/Model/BlockCache.php on line 108
[Mon Aug 17 19:41:12.706529 2020] [php7:error] [pid 17146:tid 140670127806208] [client 127.0.0.1:40702] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/vendor/magento/module-csp/Model/BlockCache.php on line 108
[Mon Aug 17 19:41:12.740424 2020] [php7:error] [pid 17146:tid 140670127806208] [client 127.0.0.1:40702] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 122880 bytes) in Unknown on line 0
[Mon Aug 17 19:41:24.887257 2020] [php7:error] [pid 16939:tid 140670111020800] [client 127.0.0.1:40798] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 4096 bytes) in /var/www/html/vendor/magento/module-csp/Model/BlockCache.php on line 78
[Mon Aug 17 19:41:24.913106 2020] [php7:error] [pid 16939:tid 140670111020800] [client 127.0.0.1:40798] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 65536 bytes) in Unknown on line 0
[Mon Aug 17 19:41:39.997127 2020] [php7:error] [pid 16933:tid 140670152984320] [client 127.0.0.1:40904] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/vendor/magento/module-csp/Model/BlockCache.php on line 108, referer: http://ip-of-my-ec2-instance:80/
[Mon Aug 17 19:41:40.030659 2020] [php7:error] [pid 16933:tid 140670152984320] [client 127.0.0.1:40904] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 40960 bytes) in Unknown on line 0, referer: http://ip-of-my-ec2-instance:80/
[Mon Aug 17 19:42:08.849538 2020] [php7:error] [pid 17146:tid 140670111020800] [client 127.0.0.1:41016] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/vendor/magento/framework/Serialize/Serializer/Json.php on line 42
[Mon Aug 17 19:42:08.865389 2020] [php7:error] [pid 17146:tid 140670111020800] [client 127.0.0.1:41016] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 32768 bytes) in Unknown on line 0
[Mon Aug 17 19:44:23.245570 2020] [php7:error] [pid 17146:tid 140670027093760] [client 127.0.0.1:41798] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/vendor/magento/framework/Serialize/Serializer/Json.php on line 42, referer: https://www.google.com/
[Mon Aug 17 19:44:23.312177 2020] [php7:error] [pid 17146:tid 140670027093760] [client 127.0.0.1:41798] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 45056 bytes) in Unknown on line 0, referer: https://www.google.com/
[Mon Aug 17 19:44:29.224653 2020] [php7:error] [pid 16939:tid 140670052271872] [client 127.0.0.1:41742] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/vendor/magento/module-csp/Model/BlockCache.php on line 108
[Mon Aug 17 19:44:29.264725 2020] [php7:error] [pid 16939:tid 140670052271872] [client 127.0.0.1:41742] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 90112 bytes) in Unknown on line 0
[Mon Aug 17 19:44:34.254009 2020] [php7:error] [pid 17146:tid 140670043879168] [client 127.0.0.1:41688] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/vendor/magento/module-csp/Model/BlockCache.php on line 108
[Mon Aug 17 19:44:34.294439 2020] [php7:error] [pid 17146:tid 140670043879168] [client 127.0.0.1:41688] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 36864 bytes) in Unknown on line 0
[Mon Aug 17 19:44:40.650547 2020] [php7:error] [pid 17146:tid 140670010308352] [client 127.0.0.1:41882] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/vendor/magento/framework/Serialize/Serializer/Json.php on line 42
[Mon Aug 17 19:44:40.690603 2020] [php7:error] [pid 17146:tid 140670010308352] [client 127.0.0.1:41882] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 49152 bytes) in Unknown on line 0
[Mon Aug 17 19:44:49.990225 2020] [php7:error] [pid 16939:tid 140670144591616] [client 127.0.0.1:42006] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 33554440 bytes) in /var/www/html/vendor/magento/module-csp/Model/Collector/DynamicCollector.php on line 31

[m2-assistant]

Hi @onepack. Thank you for your report.
To help us process this issue please make sure that you provided the following information:

• Summary of the issue
• Information on your environment
• Steps to reproduce
• Expected and actual results

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento give me 2.4-develop instance - upcoming 2.4.x release

For more details, please, review the Magento Contributor Assistant documentation.

Please, add a comment to assign the issue: @magento I am working on this

---

• Join Magento Community Engineering Slack and ask your questions in #github channel.

⚠️ According to the Magento Contribution requirements, all issues must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.

🕙 You can find the schedule on the Magento Community Calendar page.

📞 The triage of issues happens in the queue order. If you want to speed up the delivery of your contribution, please join the Community Contributions Triage session to discuss the appropriate ticket.

🎥 You can find the recording of the previous Community Contributions Triage on the Magento Youtube Channel

✏️ Feel free to post questions/proposals/feedback related to the Community Contributions Triage process to the corresponding Slack Channel

[gwharton]

Allowed memory size of 1073741824 bytes exhausted (tried to allocate 20480 bytes) in /home/ubuntu/www/vendor/magento/module-csp/Model/BlockCache.php on line 108
Allowed memory size of 1073741824 bytes exhausted (tried to allocate 67108872 bytes) in /home/ubuntu/www/vendor/magento/module-csp/Model/Collector/DynamicCollector.php on line 31
Allowed memory size of 1073741824 bytes exhausted (tried to allocate 20480 bytes) in /home/ubuntu/www/vendor/magento/framework/Serialize/Serializer/Json.php on line 37
Allowed memory size of 792723456 bytes exhausted (tried to allocate 4096 bytes) in /home/ubuntu/www/vendor/magento/module-csp/Model/BlockCache.php on line 78

I see it trying to allocate a 67MB CSP policy!!!

I'm going to run with CSP module disabled for a while and see if that helps

[onepack]

@gwharton a quick workaround is to disable the CSP module.
This weekend I'm going what the source of this issue with the CSP load is.

[gwharton]

Seemed to be something to do with the BlockCache with the limited debugging I was able to do before throwing one's hands in the air and disabling CSP.

[m2-assistant]

Hi @AlexMaxHorkun. Thank you for working on this issue.
In order to make sure that issue has enough information and ready for development, please read and check the following instruction: 👇

• 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).
     Details

     If the issue has a valid description, the label Issue: Format is valid will be added to the issue automatically. Please, edit issue description if needed, until label Issue: Format is valid appears.
• 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue. If the report is valid, add Issue: Clear Description label to the issue by yourself.
• 3. Add Component: XXXXX label(s) to the ticket, indicating the components it may be related to.
• 4. Verify that the issue is reproducible on 2.4-develop branch
     Details

     - Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
     - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
     - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!

---

• Join Magento Community Engineering Slack and ask your questions in #github channel.

[AlexMaxHorkun]

I was not able to reproduce this locally with 1200+ products and dozens of categories. Perhaps this issue can only occur on a heavily resource limited setup. However I can see that there is a room for improvement performance wise in CSP implementation and it will be addressed in the next 2.4.x release