Support S3 compatible storage

[gaelreyrol]

Description

This PR should allow users to configure the endpoint and path style for S3 compatible storage like Minio or services like Clever Cloud Cellar.

Fixed Issues

Fixes #32114

Manual testing scenarios

Start Minio with Docker and create a bucket with s3cmd.

Setup Magento Remote storage:

bin/magento setup:config:set \
    --remote-storage-driver="aws-s3" \
    --remote-storage-endpoint="http://127.0.0.1:9000" \
    --remote-storage-region="us-east-1" \
    --remote-storage-bucket="magento-media" \
    --remote-storage-key="minioadmin" \
    --remote-storage-secret="minioadmin" \
    --no-interaction

Sync media folders:

bin/magento remote-storage:sync

Comments

As $config seems to be directly passed to S3Client, I though it did not need any specific unit or integration tests.

A PR is waiting on the devdocs : magento/devdocs#8914

[m2-assistant]

Hi @gaelreyrol. Thank you for your contribution
Here are some useful tips how you can test your changes using Magento test environment.
Add the comment under your pull request to deploy test or vanilla Magento instance:

• @magento give me test instance - deploy test instance based on PR changes
• @magento give me 2.4-develop instance - deploy vanilla Magento instance

❗ Automated tests can be triggered manually with an appropriate comment:

• @magento run all tests - run or re-run all required tests against the PR changes
• @magento run <test-build(s)> - run or re-run specific test build(s)
  For example: @magento run Unit Tests

<test-build(s)> is a comma-separated list of build names. Allowed build names are:

1. Database Compare
2. Functional Tests CE
3. Functional Tests EE,
4. Functional Tests B2B
5. Integration Tests
6. Magento Health Index
7. Sample Data Tests CE
8. Sample Data Tests EE
9. Sample Data Tests B2B
10. Static Tests
11. Unit Tests
12. WebAPI Tests
13. Semantic Version Checker

You can find more information about the builds here

ℹ️ Please run only needed test builds instead of all when developing. Please run all test builds before sending your PR for review.

For more details, please, review the Magento Contributor Guide documentation.

⚠️ According to the Magento Contribution requirements, all Pull Requests must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.

🕙 You can find the schedule on the Magento Community Calendar page.

📞 The triage of Pull Requests happens in the queue order. If you want to speed up the delivery of your contribution, please join the Community Contributions Triage session to discuss the appropriate ticket.

🎥 You can find the recording of the previous Community Contributions Triage on the Magento Youtube Channel

✏️ Feel free to post questions/proposals/feedback related to the Community Contributions Triage process to the corresponding Slack Channel

[gaelreyrol]

@magento give me test instance

[magento-deployment-service]

Hi @gaelreyrol. Thank you for your request. I'm working on Magento instance for you.

[magento-deployment-service]

Hi @gaelreyrol, here is your Magento Instance: https://0ccad1232b70dd0542afe8ebd29e0193.instances.magento-community.engineering
Admin access: https://0ccad1232b70dd0542afe8ebd29e0193.instances.magento-community.engineering/admin_a592
Login: 0af0d765
Password: 3e43642034b2

[gaelreyrol]

@magento run Unit Tests

[magento-automated-testing]

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please re-request them if they don't show in a reasonable amount of time.

[gaelreyrol]

@magento run all tests

[magento-automated-testing]

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please re-request them if they don't show in a reasonable amount of time.

[slavvka]

@magento run all tests

[magento-automated-testing]

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please re-request them if they don't show in a reasonable amount of time.

[mrtuvn]

@magento run WebAPI Tests, Unit Tests

[magento-automated-testing]

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please re-request them if they don't show in a reasonable amount of time.

[engcom-Hotel]

✔️ QA Passed

Hello @gaelreyrol,

We have tested this PR with localstack with following steps:

1. Setup localstack in local system.
2. Create S3 bucket using awslocal cli.
3. Enable remote storage in Magento using below command:
   bin/magento setup:config:set --remote-storage-driver="aws-s3" --remote-storage-bucket="mg32937" --remote-storage-region="us-east-1" --remote-storage-endpoint="http://0.0.0.0:4566"
4. Sync media files in localstack's S3 storage.
5. Verify if files are saving in a local S3 bucket.

Thanks for the contribution.

[barryvdh]

Can't you better use yes/no field for the path style?

[gaelreyrol]

Can't you better use yes/no field for the path style?

I used 1/0 to stay consistent with the other options of the cli.

[barryvdh]

Yeah but it's cast to boolean anyway.

[gaelreyrol]

Yeah but it's cast to boolean anyway.

I think it's preferable to stay consistent about the usage of the cli 😉

[m2-assistant]

Hi @gaelreyrol, thank you for your contribution!
Please, complete Contribution Survey, it will take less than a minute.
Your feedback will help us to improve contribution process.

[Nuranto]

Hello,

Thank you for this MR ! Do we know if this will be released in the Feature Releases of January 18 ? Or will it be for 2.4.4 in March ?

[zapotocnylubos]

Hello, I am testing this with local MinIO docker service (php service -> minio service).

I have also tested my credentials via manual commands with postman and everything works ok, but

my installation command fails with this error

Adapter error: Unable to write file at location: storage.flag.

My installation command looks like this:

bin/magento setup:install --db-host=mysql --db-name=magento --db-user=user --db-password=password --remote-storage-driver="aws-s3" --remote-storage-endpoint="http://minio:9000/" --remote-storage-bucket="magento" --remote-storage-region="us-east-1" --remote-storage-key="<key>" --remote-storage-secret="<secret>" --remote-storage-path-style=1

Even installing magento first and then setting config value results in the same error

bin/magento setup:config:set --remote-storage-driver="aws-s3" --remote-storage-endpoint="http://minio:9000/" --remote-storage-bucket="magento" --remote-storage-region="us-east-1" --remote-storage-key="<key>" --remote-storage-secret="<secret>" --remote-storage-path-style=1

I did some digging and during setting these config values there is a routine, that tries to create this file, but url, that driver constructs at

magento2/app/code/Magento/AwsS3/Driver/AwsS3Factory.php

Line 136 in 9ef88b4

$objectUrl = rtrim($client->getObjectUrl($config['bucket'], './'), '/') . trim($prefix, '\\/') . '/';

results in this objecturl http://magento.minio:9000/storage.flag.

Even though I enabled storage path style. So correct url should be http://minio:9000/magento/storage.flag

magento.minio is of course not reachable/resolvable.

Is this correct behaviour? (how can I fix this?)
Thank you

[gaelreyrol]

Hi @zapotocnylubos,

You really should create a separate issue and not comment a PR that from what I understand does not have much to do with your problem.

You can of course link this PR if you really think it is related.

Regards,

[Nuranto]

Hi @zapotocnylubos,

There is an issue with path style parameter in this MR.
We had to patch this manually.
Have a look at my comment in "Files changed" tab.

Here is the patch :

diff --git a/Driver/AwsS3Factory.php b/Driver/AwsS3Factory.php
index 917bb6ce045e..66c95e97ace0 100644
--- a/Driver/AwsS3Factory.php
+++ b/Driver/AwsS3Factory.php
@@ -116,8 +116,8 @@
             $config['http_handler'] = $this->objectManager->create($config['http_handler'])($config);
         }
 
-        if (!empty($config['path_style'])) {
-            $config['use_path_style_endpoint'] = boolval($config['path_style']);
+        if (!empty($config['path-style'])) {
+            $config['use_path_style_endpoint'] = boolval($config['path-style']);
         }
 
         $client = new S3Client($config);