Random deadlocks during setting email sent flag

[ihor-sviziev]

During Black Friday we found random deadlocks during trying to send new order emails.
Analyzing of these cases a bit not obvious - Magento actually saves whole order object, and it saves all the order items JUST FOR setting flag that email was sent for the order. As result under high load we're having deadlocks like this:

SQLSTATE[40001]: Serialization failure: 1213 Deadlock found when trying to get lock; try restarting transaction, query was: UPDATE `sales_order_item` SET `order_id` = ?, `parent_item_id` = ?, `quote_item_id` = ?, `store_id` = ?, `created_at` = '2020-11-28 23:33:44', `product_id` = ?, `product_type` = ?, `product_options` = ?, `weight` = ?, `is_virtual` = ?, `sku` = ?, `name` = ?, `description` = ?, `applied_rule_ids` = ?, `additional_data` = ?, `is_qty_decimal` = ?, `no_discount` = ?, `qty_backordered` = ?, `qty_canceled` = ?, `qty_invoiced` = ?, `qty_ordered` = ?, `qty_refunded` = ?, `qty_shipped` = ?, `base_cost` = ?, `price` = ?, `base_price` = ?, `original_price` = ?, `base_original_price` = ?, `tax_percent` = ?, `tax_amount` = ?, `base_tax_amount` = ?, `tax_invoiced` = ?, `base_tax_invoiced` = ?, `discount_percent` = ?, `discount_amount` = ?, `base_discount_amount` = ?, `discount_invoiced` = ?, `base_discount_invoiced` = ?, `amount_refunded` = ?, `base_amount_refunded` = ?, `row_total` = ?, `base_row_total` = ?, `row_invoiced` = ?, `base_row_invoiced` = ?, `row_weight` = ?, `base_tax_before_discount` = ?, `tax_before_discount` = ?, `ext_order_item_id` = ?, `locked_do_invoice` = ?, `locked_do_ship` = ?, `price_incl_tax` = ?, `base_price_incl_tax` = ?, `row_total_incl_tax` = ?, `base_row_total_incl_tax` = ?, `discount_tax_compensation_amount` = ?, `base_discount_tax_compensation_amount` = ?, `discount_tax_compensation_invoiced` = ?, `base_discount_tax_compensation_invoiced` = ?, `discount_tax_compensation_refunded` = ?, `base_discount_tax_compensation_refunded` = ?, `tax_canceled` = ?, `discount_tax_compensation_canceled` = ?, `tax_refunded` = ?, `base_tax_refunded` = ?, `discount_refunded` = ?, `base_discount_refunded` = ?, `free_shipping` = ?, `gift_message_id` = ?, `gift_message_available` = ?, `weee_tax_applied` = ?, `weee_tax_applied_amount` = ?, `weee_tax_applied_row_amount` = ?, `weee_tax_disposition` = ?, `weee_tax_row_disposition` = ?, `base_weee_tax_applied_amount` = ?, `base_weee_tax_applied_row_amnt` = ?, `base_weee_tax_disposition` = ?, `base_weee_tax_row_disposition` = ?, `osc_gift_wrap_amount` = ?, `base_osc_gift_wrap_amount` = ?, `mp_reward_earn` = ?, `mp_reward_spent` = ?, `mp_reward_base_discount` = ?, `mp_reward_discount` = ? WHERE (item_id='1644918')

The stack trace was following:

/path/to/magento2/vendor/magento/framework/DB/Adapter/Pdo/Mysql.php:613
/path/to/magento2/vendor/magento/zendframework1/library/Zend/Db/Adapter/Abstract.php:635
/path/to/magento2/vendor/magento/module-sales/Model/ResourceModel/EntityAbstract.php:182
/path/to/magento2/vendor/magento/framework/Model/ResourceModel/Db/AbstractDb.php:419
/path/to/magento2/vendor/magento/module-sales/Model/Order/ItemRepository.php:176
/path/to/magento2/vendor/magento/module-sales/Model/ResourceModel/Order/Relation.php:74
/path/to/magento2/vendor/magento/framework/Model/ResourceModel/Db/VersionControl/RelationComposite.php:48
/path/to/magento2/vendor/magento/framework/Model/ResourceModel/Db/VersionControl/AbstractDb.php:57
/path/to/magento2/vendor/magento/framework/Model/ResourceModel/Db/AbstractDb.php:424
/path/to/magento2/vendor/magento/module-sales/Model/ResourceModel/Order.php:178
/path/to/magento2/vendor/magento/module-sales/Model/EmailSenderHandler.php:113
/path/to/magento2/vendor/magento/module-sales/Cron/SendEmails.php:46
/path/to/magento2/vendor/magento/module-cron/Observer/ProcessCronQueueObserver.php:350
/path/to/magento2/vendor/magento/module-cron/Observer/ProcessCronQueueObserver.php:809
/path/to/magento2/vendor/magento/module-cron/Observer/ProcessCronQueueObserver.php:776
/path/to/magento2/vendor/magento/module-cron/Observer/ProcessCronQueueObserver.php:267
/path/to/magento2/vendor/magento/framework/Event/Invoker/InvokerDefault.php:88
/path/to/magento2/vendor/magento/framework/Event/Invoker/InvokerDefault.php:74
/path/to/magento2/vendor/magento/framework/Event/Manager.php:66
/path/to/magento2/generated/code/Magento/Framework/Event/Manager/Proxy.php:95
/path/to/magento2/vendor/magento/framework/App/Cron.php:86
/path/to/magento2/vendor/magento/module-cron/Console/Command/CronCommand.php:117
/path/to/magento2/vendor/symfony/console/Command/Command.php:255
/path/to/magento2/vendor/magento/framework/Interception/Interceptor.php:58
/path/to/magento2/vendor/magento/framework/Interception/Interceptor.php:138
/path/to/magento2/vendor/magento/framework/Interception/Interceptor.php:153
/path/to/magento2/generated/code/Magento/Cron/Console/Command/CronCommand/Interceptor.php:26
/path/to/magento2/vendor/symfony/console/Application.php:1005
/path/to/magento2/vendor/symfony/console/Application.php:271
/path/to/magento2/vendor/magento/framework/Console/Cli.php:115
/path/to/magento2/vendor/symfony/console/Application.php:147
/path/to/magento2/bin/magento:23

Potentially such behavior might cause issues with double sending new order emails.

Preconditions (*)

1. Magento 2.3.6, 2.4-develop
2. Enabled async email sending

Steps to reproduce (*)

1. Disable cron jobs
2. Place an order
3. Set breakpoint at

   magento2/app/code/Magento/Sales/Model/EmailSenderHandler.php

   Line 111 in d7ab99d

   if ($this->emailSender->send($item, true)) {

   and

   magento2/app/code/Magento/Sales/Model/ResourceModel/Order/Relation.php

   Line 69 in d7ab99d

   if (null !== $object->getItems()) {

4. Run php bin/magento cron:run

Expected result (*)

1. We should have just tiny SQL for updating one column, not the whole object and all related objects

• so our debug should stop only on 1st breakpoint

Actual result (*)

1. We have saving of whole object and all their related objects that causing deadlocks under the load

• so our debug stopping on 1st and 2nd breakpoints

---

Please provide Severity assessment for the Issue as Reporter. This information will help during Confirmation and Issue triage processes.

• Severity: S0 - Affects critical data or functionality and leaves users without workaround.
• Severity: S1 - Affects critical data or functionality and forces users to employ a workaround.
• Severity: S2 - Affects non-critical data or functionality and forces users to employ a workaround.
• Severity: S3 - Affects non-critical data or functionality and does not force users to employ a workaround.
• Severity: S4 - Affects aesthetics, professional look and feel, “quality” or “usability”.

[m2-assistant]

Hi @ihor-sviziev. Thank you for your report.
To help us process this issue please make sure that you provided the following information:

• Summary of the issue
• Information on your environment
• Steps to reproduce
• Expected and actual results

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento give me 2.4-develop instance - upcoming 2.4.x release

For more details, please, review the Magento Contributor Assistant documentation.

Please, add a comment to assign the issue: @magento I am working on this

---

• Join Magento Community Engineering Slack and ask your questions in #github channel.

⚠️ According to the Magento Contribution requirements, all issues must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.

🕙 You can find the schedule on the Magento Community Calendar page.

📞 The triage of issues happens in the queue order. If you want to speed up the delivery of your contribution, please join the Community Contributions Triage session to discuss the appropriate ticket.

🎥 You can find the recording of the previous Community Contributions Triage on the Magento Youtube Channel

✏️ Feel free to post questions/proposals/feedback related to the Community Contributions Triage process to the corresponding Slack Channel

[pmathbliss]

I'm seeing something similar to this with a process that regularly creates 50-100 invoices an hour.

[magento-engcom-team]

@gabrieldagama Thank you for verifying the issue.

Unfortunately, not enough information was provided to acknowledge ticket. Please consider adding the following:

• Add "Reproduced on " label(s) to this ticket based on verification result

Once all required information is added, please add label "Issue: Confirmed" again.
Thanks!

[magento-engcom-team]

@gabrieldagama Thank you for verifying the issue.

Unfortunately, not enough information was provided to acknowledge ticket. Please consider adding the following:

• Add "Reproduced on " label(s) to this ticket based on verification result

Once all required information is added, please add label "Issue: Confirmed" again.
Thanks!