Support batches processing for media gallery synchronization queue messages #29543

yolouiese · 2020-08-14T13:01:42Z

Description (*)

Added an implementation to support batch processing for synchronization queue messages

Related Pull Requests

https://github.com/magento/partners-magento2-infrastructure/pull/36

Fixed Issues (if relevant)

Fixes Support batches processing for synchronization queue messages adobe-stock-integration#1724: Support batches processing for synchronization queue messages

Manual testing scenarios (*)

Check synchronization when enabling new media gallery
Check bin/magento media-gallery:sync functionality
Check bin/magento media-content:sync functionality

General synchronization test case:

Upload a couple of images or save images from Adobe Stock
Add/insert images to either category, product, cms pages or blocks
Delete rows from tables media_gallery_asset and media_content_asset
Disable Enhanced Media Gallery and save configuration (Assuming it is already enabled)
Enable Enhanced Media Gallery and save configuration (Changes in the system configuration for the Enhanced Media Gallery triggers the Media Gallery and Media Content sychronization publishers)
Run bin/magento queue:consumers:start media.gallery.synchronization
Open Media Gallery and ensure the images are present in the grid. (Check media_gallery_asset table if data are saved)
Run bin/magento queue:consumers:start media.content.synchronization
View details of the images in the media gallery and ensure the Used In information is correct. (Check media_content_asset table if data are saved)

Batch synchronization test case:

Upload a couple of images or save images from Adobe Stock
Add/insert these images to either category, product, cms pages or blocks
Delete rows from tables media_gallery_asset and media_content_asset
Call the \Magento\MediaGallerySynchronization\Model\Publish::execute passing the paths of the images
Run bin/magento queue:consumers:start media.gallery.synchronization
Open Media Gallery and ensure the images are present in the grid. (Check media_gallery_asset table if data are saved)
Call the \Magento\MediaContentSynchronization\Model\Publish::execute passing the contentIdentities of the places where the images have been added to
Run bin/magento queue:consumers:start media.content.synchronization
View details of the images in the media gallery and ensure the Used In information is correct. (Check media_content_asset table if data are saved)

Questions or comments

Contribution checklist (*)

Pull request has a meaningful description of its purpose
All commits are accompanied by meaningful commit messages
All new or changed code is covered with unit/integration tests (if applicable)
All automated tests passed successfully (all builds are green)

…ssages - updated synchronization consumer and added media content bulk service

m2-assistant · 2020-08-14T13:01:45Z

Hi @yolouiese. Thank you for your contribution
Here is some useful tips how you can test your changes using Magento test environment.
Add the comment under your pull request to deploy test or vanilla Magento instance:

@magento give me test instance - deploy test instance based on PR changes
@magento give me 2.4-develop instance - deploy vanilla Magento instance

❗ Automated tests can be triggered manually with an appropriate comment:

@magento run all tests - run or re-run all required tests against the PR changes
@magento run <test-build(s)> - run or re-run specific test build(s)
For example: @magento run Unit Tests

<test-build(s)> is a comma-separated list of build names. Allowed build names are:

Database Compare
Functional Tests CE
Functional Tests EE,
Functional Tests B2B
Integration Tests
Magento Health Index
Sample Data Tests CE
Sample Data Tests EE
Sample Data Tests B2B
Static Tests
Unit Tests
WebAPI Tests

You can find more information about the builds here

ℹ️ Please run only needed test builds instead of all when developing. Please run all test builds before sending your PR for review.

For more details, please, review the Magento Contributor Guide documentation.

⚠️ According to the Magento Contribution requirements, all Pull Requests must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.

🕙 You can find the schedule on the Magento Community Calendar page.

📞 The triage of Pull Requests happens in the queue order. If you want to speed up the delivery of your contribution, please join the Community Contributions Triage session to discuss the appropriate ticket.

🎥 You can find the recording of the previous Community Contributions Triage on the Magento Youtube Channel

✏️ Feel free to post questions/proposals/feedback related to the Community Contributions Triage process to the corresponding Slack Channel

tanghongtanghong · 2020-08-17T06:40:38Z

hi,hello

…upport-batches-synchronization

…synchronization queue messages - implemented gallery synchronization functionality

…upport-batches-synchronization

…synchronization queue messages - implemented media content sychronization

yolouiese · 2020-08-20T20:07:22Z

@magento run all tests

yolouiese · 2020-08-20T23:25:17Z

@magento run all tests

…synchronization queue messages - added media content sychronizer and save contents to database

…upport-batches-synchronization

yolouiese · 2020-08-21T11:46:27Z

@magento run all tests

…synchronization queue messages - added method to sync and save media content to database

yolouiese · 2020-08-21T15:36:13Z

@magento run all tests

…synchronization queue messages - fixed failed static test

yolouiese · 2020-08-21T17:39:27Z

@magento run all tests

…synchronization queue messages - fixed failed static test

…upport-batches-synchronization

yolouiese · 2020-09-03T18:39:49Z

@magento run all tests

sivaschenko · 2020-09-06T10:08:52Z

@magento run all tests

sivaschenko · 2020-09-06T10:11:16Z

app/code/Magento/MediaContentSynchronization/Model/SynchronizeIdentities.php

+            );
+        }
+
+        $this->setLastExecutionTime();


The falg should be set only after full synchronization, not after partial

sivaschenko · 2020-09-06T10:17:43Z

app/code/Magento/MediaContentSynchronization/Model/SynchronizeIdentities.php

+        }
+
+        $this->setLastExecutionTime();
+        $this->removeObsoleteContent->execute();


RemoveObsoleteContentAsset is a part of full synchronization, shouldn't be a part of partial

sivaschenko · 2020-09-06T10:23:42Z

...code/Magento/MediaContentSynchronizationCatalog/Model/Synchronizer/SynchronizeIdentities.php

+    /**
+     * @inheritDoc
+     */
+    public function execute(array $mediaContentIdentities): void


It would be good to check that the content links are removed if the entity is no longer present (was deleted)

sivaschenko · 2020-09-06T10:36:47Z

@magento run all tests

magento-engcom-team · 2020-09-06T10:37:42Z

Hi @sivaschenko, thank you for the review.
ENGCOM-8146 has been created to process this Pull Request

sivaschenko · 2020-09-10T10:21:57Z

@magento run Integration Tests

sivaschenko · 2020-09-10T10:23:06Z

@magento run all tests

…tion queue messages #29543

m2-assistant · 2020-09-11T11:03:10Z

Hi @yolouiese, thank you for your contribution!
Please, complete Contribution Survey, it will take less than a minute.
Your feedback will help us to improve contribution process.

magento#1724: Support batches processing for synchronization queue me…

b6a66b3

…ssages - updated synchronization consumer and added media content bulk service

yolouiese requested a review from sivaschenko August 14, 2020 13:01

ghost added the Progress: pending review label Aug 14, 2020

magento-engcom-team added Component: MediaContentSynchronizationApi Component: MediaGallerySynchronization Component: MediaContentSynchronization Release Line: 2.4 labels Aug 14, 2020

yolouiese added the Project: ASI label Aug 14, 2020

tanghongtanghong approved these changes Aug 17, 2020

View reviewed changes

yolouiese added 2 commits August 19, 2020 02:05

Merge branch '2.4-develop' of github.com:magento/magento2 into 1724-s…

a550e1e

…upport-batches-synchronization

magento/adobe-stock-integration#1724: Support batches processing for …

29c3b26

…synchronization queue messages - implemented gallery synchronization functionality

sivaschenko added Priority: P3 May be fixed according to the position in the backlog. Severity: S3 Affects non-critical data or functionality and does not force users to employ a workaround. labels Aug 19, 2020

yolouiese added 2 commits August 21, 2020 03:56

Merge branch '2.4-develop' of github.com:magento/magento2 into 1724-s…

7bacf97

…upport-batches-synchronization

magento/adobe-stock-integration#1724: Support batches processing for …

bbb99e5

…synchronization queue messages - implemented media content sychronization

yolouiese added 2 commits August 21, 2020 07:26

magento/adobe-stock-integration#1724: Support batches processing for …

2917c4b

…synchronization queue messages - added media content sychronizer and save contents to database

Merge branch '2.4-develop' of github.com:magento/magento2 into 1724-s…

d98fcf3

…upport-batches-synchronization

magento/adobe-stock-integration#1724: Support batches processing for …

16dc1fa

…synchronization queue messages - added method to sync and save media content to database

magento/adobe-stock-integration#1724: Support batches processing for …

d5276dd

…synchronization queue messages - fixed failed static test

yolouiese added 2 commits August 22, 2020 01:40

magento/adobe-stock-integration#1724: Support batches processing for …

d9e9075

…synchronization queue messages - fixed failed static test

Merge branch '2.4-develop' of github.com:magento/magento2 into 1724-s…

bf3c03a

…upport-batches-synchronization

sivaschenko self-assigned this Aug 24, 2020

Merge branch '2.4-develop' of github.com:magento/magento2 into 1724-s…

56ed88f

…upport-batches-synchronization

sivaschenko added the Auto-Tests: Covered All changes in Pull Request is covered by auto-tests label Sep 6, 2020

Merge branch '2.4-develop' into 1724-support-batches-synchronization

f221320

sivaschenko reviewed Sep 6, 2020

View reviewed changes

magento#29543: Removed general operations from partial synchronizer

ee1d2d2

sivaschenko approved these changes Sep 6, 2020

View reviewed changes

magento-engcom-team added Component: MediaContentSynchronizationCatalog Component: MediaContentSynchronizationCms labels Sep 6, 2020

ghost added Progress: ready for testing and removed Progress: needs update labels Sep 6, 2020

engcom-Bravo self-assigned this Sep 8, 2020

ghost added Progress: testing in progress Progress: extended testing and removed Progress: ready for testing Progress: testing in progress labels Sep 8, 2020

engcom-Bravo added Progress: testing in progress QA: Added to Regression Scope Scenario was analysed and added to Regression Testing Scope and removed Progress: extended testing labels Sep 9, 2020

Merge branch '2.4-develop' into 1724-support-batches-synchronization

01a968e

ghost added Progress: accept and removed Progress: testing in progress labels Sep 10, 2020

gabrieldagama mentioned this pull request Sep 11, 2020

Issues with registering custom UI components. #1724

Closed

magento-engcom-team pushed a commit that referenced this pull request Sep 11, 2020

ENGCOM-8146: Support batches processing for media gallery synchroniza…

b9aab1a

…tion queue messages #29543

magento-engcom-team merged commit c8e7c8d into magento:2.4-develop Sep 11, 2020

Support batches processing for media gallery synchronization queue messages #29543

Support batches processing for media gallery synchronization queue messages #29543

Uh oh!

Conversation

yolouiese commented Aug 14, 2020 • edited by sivaschenko Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description (*)

Related Pull Requests

Fixed Issues (if relevant)

Manual testing scenarios (*)

Questions or comments

Contribution checklist (*)

Uh oh!

m2-assistant bot commented Aug 14, 2020

Uh oh!

tanghongtanghong commented Aug 17, 2020

Uh oh!

yolouiese commented Aug 20, 2020

Uh oh!

yolouiese commented Aug 20, 2020

Uh oh!

yolouiese commented Aug 21, 2020

Uh oh!

yolouiese commented Aug 21, 2020

Uh oh!

yolouiese commented Aug 21, 2020

Uh oh!

yolouiese commented Sep 3, 2020

Uh oh!

sivaschenko commented Sep 6, 2020

Uh oh!

sivaschenko Sep 6, 2020

Choose a reason for hiding this comment

Uh oh!

sivaschenko Sep 6, 2020

Choose a reason for hiding this comment

Uh oh!

sivaschenko Sep 6, 2020

Choose a reason for hiding this comment

Uh oh!

sivaschenko commented Sep 6, 2020

Uh oh!

magento-engcom-team commented Sep 6, 2020

Uh oh!

sivaschenko commented Sep 10, 2020

Uh oh!

sivaschenko commented Sep 10, 2020

Uh oh!

m2-assistant bot commented Sep 11, 2020

Uh oh!

Uh oh!

yolouiese commented Aug 14, 2020 •

edited by sivaschenko

Loading