Greatly improve performance by caching subsequent glob calls #33998

thlassche · 2021-09-06T07:52:36Z

Description (*)

This pull request introduces an in-memory cache for glob system calls. On systems where file I/O is slow (for example Docker for Mac), glob system calls can be quite slow. This pull requests caches the same glob calls and returns a cached result if possible. In my testing this reduced the glob calls on the category listings from ~8K to ~1.6K.

Blackfire comparison:

Related Pull Requests

None

Fixed Issues (if relevant)

None

Fixes magento/magento2#<issue_number>

Manual testing scenarios (*)

Profile any page before this diff is applied
Apply diff and reprofile
See a reduction in glob() calls

Questions or comments

Contribution checklist (*)

Pull request has a meaningful description of its purpose
All commits are accompanied by meaningful commit messages
All new or changed code is covered with unit/integration tests (if applicable)
README.md files for modified modules are updated and included in the pull request if any README.md predefined sections require an update
All automated tests passed successfully (all builds are green)

Resolved issues:

resolves [Issue] Greatly improve performance by caching subsequent glob calls #34024: Greatly improve performance by caching subsequent glob calls

Resolved issues:

resolves [Issue] Greatly improve performance by caching subsequent glob calls #34025: Greatly improve performance by caching subsequent glob calls

m2-assistant · 2021-09-06T07:54:08Z

Hi @thlassche, thank you for your contribution!
Please, complete Contribution Survey, it will take less than a minute.
Your feedback will help us to improve contribution process.

m2-assistant · 2021-09-06T07:54:11Z

Hi @thlassche. Thank you for your contribution
Here are some useful tips how you can test your changes using Magento test environment.
Add the comment under your pull request to deploy test or vanilla Magento instance:

@magento give me test instance - deploy test instance based on PR changes
@magento give me 2.4-develop instance - deploy vanilla Magento instance

❗ Automated tests can be triggered manually with an appropriate comment:

@magento run all tests - run or re-run all required tests against the PR changes
@magento run <test-build(s)> - run or re-run specific test build(s)
For example: @magento run Unit Tests

<test-build(s)> is a comma-separated list of build names. Allowed build names are:

Database Compare
Functional Tests CE
Functional Tests EE,
Functional Tests B2B
Integration Tests
Magento Health Index
Sample Data Tests CE
Sample Data Tests EE
Sample Data Tests B2B
Static Tests
Unit Tests
WebAPI Tests
Semantic Version Checker

You can find more information about the builds here

ℹ️ Please run only needed test builds instead of all when developing. Please run all test builds before sending your PR for review.

For more details, please, review the Magento Contributor Guide documentation.

⚠️ According to the Magento Contribution requirements, all Pull Requests must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.

🕙 You can find the schedule on the Magento Community Calendar page.

📞 The triage of Pull Requests happens in the queue order. If you want to speed up the delivery of your contribution, please join the Community Contributions Triage session to discuss the appropriate ticket.

🎥 You can find the recording of the previous Community Contributions Triage on the Magento Youtube Channel

✏️ Feel free to post questions/proposals/feedback related to the Community Contributions Triage process to the corresponding Slack Channel

ihor-sviziev · 2021-09-06T08:05:37Z

@magento run all tests

magento-automated-testing · 2021-09-06T08:05:43Z

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please re-request them if they don't show in a reasonable amount of time.

lib/internal/Magento/Framework/Filesystem/Glob.php

Co-authored-by: Ihor Sviziev <[email protected]>

thlassche · 2021-09-06T08:18:58Z

@magento run all tests

magento-automated-testing · 2021-09-06T08:19:05Z

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please re-request them if they don't show in a reasonable amount of time.

ihor-sviziev · 2021-09-06T09:00:56Z

It might significantly improve performance not only on dockerized environments but also on the ec2 instances or any server using network file storage as magento storage.

magento-engcom-team · 2021-09-06T13:21:28Z

Hi @andrewbess, thank you for the review.
ENGCOM-9214 has been created to process this Pull Request
✳️ @andrewbess, could you please add one of the following labels to the Pull Request?

Label	Description
`Auto-Tests: Covered`	All changes in Pull Request is covered by auto-tests
`Auto-Tests: Not Covered`	Changes in Pull Request requires coverage by auto-tests
`Auto-Tests: Not Required`	Changes in Pull Request does not require coverage by auto-tests

magento-engcom-team · 2021-09-06T14:17:27Z

Hi @ihor-sviziev, thank you for the review.
ENGCOM-9214 has been created to process this Pull Request
✳️ @ihor-sviziev, could you please add one of the following labels to the Pull Request?

Label	Description
`Auto-Tests: Covered`	All changes in Pull Request is covered by auto-tests
`Auto-Tests: Not Covered`	Changes in Pull Request requires coverage by auto-tests
`Auto-Tests: Not Required`	Changes in Pull Request does not require coverage by auto-tests

ihor-sviziev · 2021-09-06T14:20:26Z

I think we don't need to cover it with any test as it will be not so easy.

engcom-Hotel · 2021-09-08T12:51:11Z

@magento run Functional Tests B2B, Functional Tests CE, Functional Tests EE

magento-automated-testing · 2021-09-08T12:51:15Z

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please re-request them if they don't show in a reasonable amount of time.

onlinebizsoft · 2021-09-08T14:34:18Z

It might significantly improve performance not only on dockerized environments but also on the ec2 instances or any server using network file storage as magento storage.

What do you mean by "magento storage"? You mean "media storage"? @ihor-sviziev

ihor-sviziev · 2021-09-08T14:37:05Z

@onlinebizsoft, I think it's not only media folder, but also where magento code is located.

engcom-Hotel

Hello @thlassche,

Some tests are failing for e.g. Functional Tests B2B, Functional Tests CE, Functional Tests EE. Please look into this.

ihor-sviziev · 2021-09-08T14:42:14Z

@engcom-Hotel, the service that responsible for the currency API doesn't work. Here is confirmation: #34021 (comment)

onlinebizsoft · 2021-09-08T14:45:04Z

@onlinebizsoft, I think it's not only media folder, but also where magento code is located.

Interesting! Im not sure how glob is used and called in Magento. I thought Magento files would be loaded in OPcache memory already

engcom-Hotel · 2021-09-08T14:52:04Z

@engcom-Hotel, the service that responsible for the currency API doesn't work. Here is confirmation: #34021 (comment)

Thanks for the clarification @ihor-sviziev

engcom-Hotel · 2021-09-08T15:00:05Z

✔️ QA Passed

Hello @thlassche,

Thanks for the contribution!

Preconditions
Magento 2.4-develop branch

Manual testing scenario

Run profiler on Magento 2.4-develop with sample data installed.
Run profiler on this PR fix with sample data installed.
Compare results.

onlinebizsoft · 2021-09-10T14:31:12Z

✔️ QA Passed

Hello @thlassche,

Thanks for the contribution!

Preconditions
Magento 2.4-develop branch

Manual testing scenario
1. Run profiler on Magento 2.4-develop with sample data installed.

2. Run profiler on this PR fix with sample data installed.

3. Compare results.

Just reduce few ms?!!

thlassche · 2021-09-10T14:37:16Z

✔️ QA Passed

Hello @thlassche,

Thanks for the contribution!

Preconditions

Magento 2.4-develop branch

Manual testing scenario
1. Run profiler on Magento 2.4-develop with sample data installed.
2. Run profiler on this PR fix with sample data installed.
3. Compare results.
Just reduce few ms?!!

It depends a lot on what is currently in the cache and what is not. The first load of uncached pages will benefit way more.

ihor-sviziev · 2021-09-20T11:21:55Z

@magento run Functional Tests B2B, Functional Tests CE, Functional Tests EE

magento-automated-testing · 2021-09-20T11:22:01Z

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please re-request them if they don't show in a reasonable amount of time.

Den4ik · 2021-09-24T14:45:07Z

@magento run all test

magento-automated-testing · 2021-09-24T14:45:12Z

Failed to run the builds. Please try to re-run them later.

Den4ik · 2021-09-24T14:45:59Z

@magento run all tests

magento-automated-testing · 2021-09-24T14:46:07Z

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please re-request them if they don't show in a reasonable amount of time.

…alls #33998

Cache subsequent glob calls

ef76b7e

magento-engcom-team added Component: Filesystem Release Line: 2.4 labels Sep 6, 2021

thlassche closed this Sep 6, 2021

thlassche reopened this Sep 6, 2021

ihor-sviziev reviewed Sep 6, 2021

View reviewed changes

lib/internal/Magento/Framework/Filesystem/Glob.php Outdated Show resolved Hide resolved

m2-community-project bot assigned ihor-sviziev Sep 6, 2021

m2-community-project bot added the Progress: review label Sep 6, 2021

Make glob cache property private

a06fcad

Co-authored-by: Ihor Sviziev <[email protected]>

andrewbess self-assigned this Sep 6, 2021

ihor-sviziev added Area: Perf/Frontend All tickets related with improving frontend performance. Severity: S1 Affects critical data or functionality and forces users to employ a workaround. Area: Performance labels Sep 6, 2021

ihor-sviziev changed the title ~~Greatly improve performance on dockerized Magento instances (mainly on mac) by caching subsequent glob calls~~ Greatly improve performance by caching subsequent glob calls Sep 6, 2021

andrewbess approved these changes Sep 6, 2021

View reviewed changes

m2-community-project bot added Progress: ready for testing and removed Progress: review labels Sep 6, 2021

ihor-sviziev approved these changes Sep 6, 2021

View reviewed changes

ihor-sviziev added the Award: category of expertise label Sep 6, 2021

ihor-sviziev added the Auto-Tests: Not Required Changes in Pull Request does not require coverage by auto-tests label Sep 6, 2021

m2-community-project bot added Progress: testing in progress and removed Progress: ready for testing labels Sep 8, 2021

engcom-Hotel requested changes Sep 8, 2021

View reviewed changes

m2-community-project bot added Progress: needs update and removed Progress: testing in progress labels Sep 8, 2021

ihor-sviziev added the Progress: ready for testing label Sep 8, 2021

m2-community-project bot removed the Progress: needs update label Sep 8, 2021

m2-community-project bot added Progress: testing in progress and removed Progress: ready for testing labels Sep 8, 2021

m2-community-project bot added Progress: accept and removed Progress: testing in progress labels Sep 8, 2021

Merge branch '2.4-develop' into cache-subsequent-globs

1a5ec6c

magento-engcom-team pushed a commit that referenced this pull request Nov 21, 2021

ENGCOM-9214: Greatly improve performance by caching subsequent glob c…

2e9afb2

…alls #33998

magento-engcom-team merged commit eae78dc into magento:2.4-develop Nov 21, 2021

Greatly improve performance by caching subsequent glob calls #33998

Greatly improve performance by caching subsequent glob calls #33998

Uh oh!

Conversation

thlassche commented Sep 6, 2021 • edited by m2-assistant bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description (*)

Related Pull Requests

Fixed Issues (if relevant)

Manual testing scenarios (*)

Questions or comments

Contribution checklist (*)

Resolved issues:

Resolved issues:

Uh oh!

m2-assistant bot commented Sep 6, 2021

Uh oh!

m2-assistant bot commented Sep 6, 2021

Uh oh!

ihor-sviziev commented Sep 6, 2021

Uh oh!

magento-automated-testing bot commented Sep 6, 2021

Uh oh!

Uh oh!

thlassche commented Sep 6, 2021

Uh oh!

magento-automated-testing bot commented Sep 6, 2021

Uh oh!

ihor-sviziev commented Sep 6, 2021

Uh oh!

magento-engcom-team commented Sep 6, 2021

Uh oh!

magento-engcom-team commented Sep 6, 2021

Uh oh!

ihor-sviziev commented Sep 6, 2021

Uh oh!

engcom-Hotel commented Sep 8, 2021

Uh oh!

magento-automated-testing bot commented Sep 8, 2021

Uh oh!

onlinebizsoft commented Sep 8, 2021

Uh oh!

ihor-sviziev commented Sep 8, 2021

Uh oh!

engcom-Hotel left a comment

Choose a reason for hiding this comment

Uh oh!

ihor-sviziev commented Sep 8, 2021

Uh oh!

onlinebizsoft commented Sep 8, 2021

Uh oh!

engcom-Hotel commented Sep 8, 2021

Uh oh!

engcom-Hotel commented Sep 8, 2021

Uh oh!

onlinebizsoft commented Sep 10, 2021

Uh oh!

thlassche commented Sep 10, 2021

Uh oh!

ihor-sviziev commented Sep 20, 2021

Uh oh!

magento-automated-testing bot commented Sep 20, 2021

Uh oh!

Den4ik commented Sep 24, 2021

Uh oh!

magento-automated-testing bot commented Sep 24, 2021

Uh oh!

Den4ik commented Sep 24, 2021

Uh oh!

magento-automated-testing bot commented Sep 24, 2021

Uh oh!

Uh oh!

thlassche commented Sep 6, 2021 •

edited by m2-assistant bot

Loading