[Reproducible Builds] Diff in generated/metadata between the builds

[ihor-sviziev]

It's really important to have Reproducible Builds. It will help to check diff between the builds, so you'll be sure that you deploy only needed changes to production.

Preconditions (*)

1. Magento 2.2.8 & 2.3.x
2. Configured Build system on any CI (Bitbucket Pipelines in my case)
3. When build created - create tar.gz or any other archive and upload it somewhere to make it availabe for downloads later (AWS S3 for instance)

Steps to reproduce (*)

1. On build system make sure that you're executing php bin/magento setup:di:compile
2. Execute build preparation multiple times for single commit on two different PCs
3. Download archives for all builds that you prepared
4. Unpack archives and compare it with diff -rq ./build-1/ ./build-2/

Expected result (*)

1. You should have exactly the same results in generated/metadata folders

Actual result (*)

1. We have diff:

Files build-1/magento2/generated/metadata/crontab.php and build-2/magento2/generated/metadata/crontab.php differ
Files build-1/magento2/generated/metadata/frontend.php and build-2/magento2/generated/metadata/frontend.php differ
Files build-1/magento2/generated/metadata/global.php and build-2/magento2/generated/metadata/global.php differ
Files build-1/magento2/generated/metadata/webapi_rest.php and build-2/magento2/generated/metadata/webapi_rest.php differ
Files build-1/magento2/generated/metadata/webapi_soap.php and build-2/magento2/generated/metadata/webapi_soap.php differ

Additional info

This issue reproducing all the time on Bitbucket Pipelines, but locally I can't reproduce it. I think it caused by using different servers on each build on CI.

If we'll try to compare these files after sorting - they actually the same. I sorted them using following script:

<?php
// Note this method returns a boolean and not the array
function recur_ksort(&$array) {
   foreach ($array as &$value) {
      if (is_array($value)) recur_ksort($value);
   }
   return ksort($array);
}
$result = include 'build-1/magento2/generated/metadata/adminhtml.php';
recur_ksort($result);

echo var_export($result);

[m2-assistant]

Hi @ihor-sviziev. Thank you for your report.
To help us process this issue please make sure that you provided the following information:

• Summary of the issue
• Information on your environment
• Steps to reproduce
• Expected and actual results

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento give me 2.3-develop instance - upcoming 2.3.x release

For more details, please, review the Magento Contributor Assistant documentation.

@ihor-sviziev do you confirm that you were able to reproduce the issue on vanilla Magento instance following steps to reproduce?

• yes
• no

---

• Join Magento Community Engineering Slack and ask your questions in #github channel.

[orlangur]

@ihor-sviziev,

Actual result (*)
We have diff:

Please explain.

using RecursiveDirectoryIterator object for generating list of all classes and all their arguments and it could return not sorted result.

This is perfectly valid. However, system behavior must not vary depending on order of generation.

https://reproducible-builds.org/ has nothing to do with this issue: it's all about no backdoors in compilers.

[ihor-sviziev]

Hi @orlangur,

Actual result (*)
We have diff:

Please explain.

Result files after DI compile should be exactly the same when you running it from one commit with the same software versions, in my case - single docker image (including image hash) was used in all cases. But result is different.

using RecursiveDirectoryIterator object for generating list of all classes and all their arguments and it could return not sorted result.

This is perfectly valid. However, system behavior must not vary depending on order of generation.

It might not vary right now, but at some point it will because of some bug will be introduced. This part isn’t covered by 100% with tests.

https://reproducible-builds.org/ has nothing to do with this issue: it's all about no backdoors in compilers.

Directly - yes, it’s not related, but if we’ll look from another angle - it’s related. you can see example from symfony framework- they’re using the same approach https://symfony.com/blog/new-in-symfony-reproducible-builds

Explained more simply in the case of Symfony: if you build the container and warm up the cache of the same unchanged application multiple times, the result should always be the same.

For magento we could say the same - if you run DI compile of the same unchanged application multiple times, the result should be the same.

[orlangur]

But result is different.

Just provide the difference. No buzz :)

[ihor-sviziev]

But result is different.

Just provide the difference. No buzz :)

Diff really huge, but just to understand - in build 1 we have

<?php return array (
  'arguments' => 
  array (
    'Aheadworks\\Giftcard\\Api\\Data\\AmountExtension' => 
    array (
      'data' => 
      array (
        '_v_' => 
        array (
        ),
      ),
    ),
    'Aheadworks\\Giftcard\\Api\\Data\\AmountExtensionFactory' => 
    array (
      'objectManager' => 
      array (
        '_i_' => 'Magento\\Framework\\ObjectManagerInterface',
      ),
      'instanceName' => 
      array (
        '_v_' => '\\Aheadworks\\Giftcard\\Api\\Data\\AmountExtension',
      ),
    ),
    'Aheadworks\\Giftcard\\Api\\Data\\AmountInterfaceFactory' => 
    array (
      'objectManager' => 
      array (
        '_i_' => 'Magento\\Framework\\ObjectManagerInterface',
      ),
      'instanceName' => 
      array (
        '_v_' => '\\Aheadworks\\Giftcard\\Api\\Data\\AmountInterface',
      ),
    ),
    'Aheadworks\\Giftcard\\Api\\Data\\CodeGenerationSettingsExtension' => 
    array (
      'data' => 
      array (
        '_v_' => 
        array (
        ),
      ),
    )
...

In build 2 we have following:

<?php return array (
  'arguments' => 
  array (
    'Aheadworks\\Giftcard\\Pricing\\Price\\ConfiguredPrice' => 
    array (
      'saleableItem' => 
      array (
        '_i_' => 'Magento\\Catalog\\Model\\Product\\Interceptor',
      ),
      'quantity' => 
      array (
        '_vn_' => true,
      ),
      'calculator' => 
      array (
        '_i_' => 'Magento\\Framework\\Pricing\\Adjustment\\Calculator',
      ),
      'priceCurrency' => 
      array (
        '_i_' => 'Magento\\Directory\\Model\\PriceCurrency',
      ),
      'item' => 
      array (
        '_vn_' => true,
      ),
      'configuredOptions' => 
      array (
        '_vn_' => true,
      ),
    ),
    'Aheadworks\\Giftcard\\Pricing\\Price\\FinalPrice' => 
    array (
      'saleableItem' => 
      array (
        '_i_' => 'Magento\\Framework\\Pricing\\SaleableInterface',
      ),
      'quantity' => 
      array (
        '_vn_' => true,
      ),
      'calculator' => 
      array (
        '_i_' => 'Magento\\Framework\\Pricing\\Adjustment\\Calculator',
      ),
      'priceCurrency' => 
      array (
        '_i_' => 'Magento\\Directory\\Model\\PriceCurrency',
      ),
    ),
    'Aheadworks\\Giftcard\\Helper\\Data' => 
    array (
      'context' => 
      array (
        '_i_' => 'Magento\\Framework\\App\\Helper\\Context',
      ),
    ),
    'Aheadworks\\Giftcard\\Observer\\ControllerActionPredispatchObserver' => 
    array (
      'session' => 
      array (
        '_i_' => 'Magento\\Backend\\Model\\Session\\Interceptor',
      ),
      'actionFlag' => 
      array (
        '_i_' => 'Magento\\Framework\\App\\ActionFlag',
      ),
      'url' => 
      array (
        '_i_' => 'Magento\\Backend\\Model\\Url\\Interceptor',
      ),
    ),
    'Aheadworks\\Giftcard\\Observer\\OrderCreationProcessDataObserver' => 
    array (
      'giftcardCartManagement' => 
      array (
        '_i_' => 'Aheadworks\\Giftcard\\Model\\Service\\GiftcardCartService',
      ),
      'messageManager' => 
      array (
        '_i_' => 'Magento\\Framework\\Message\\Manager',
      ),
      'escaper' => 
      array (
        '_i_' => 'Magento\\Framework\\Escaper',
      ),
    ),
    ...
    'Aheadworks\\Giftcard\\Api\\Data\\AmountExtensionFactory' => 
    array (
      'objectManager' => 
      array (
        '_i_' => 'Magento\\Framework\\ObjectManagerInterface',
      ),
      'instanceName' => 
      array (
        '_v_' => '\\Aheadworks\\Giftcard\\Api\\Data\\AmountExtension',
      ),
    ),
    'Aheadworks\\Giftcard\\Api\\Data\\OptionExtension' => 
    array (
      'data' => 
      array (
        '_v_' => 
        array (
        ),
      ),
    ),
    
    ...
    'Aheadworks\\Giftcard\\Api\\Data\\AmountExtension' => 
    array (
      'data' => 
      array (
        '_v_' => 
        array (
        ),
      ),
    ),

All classes are present in both files, no differences between the data, just sort order is different.

I just showed for 1 custom module, but it appears for all modules, including Magento core.

[orlangur]

@ihor-sviziev I see, just some class list in order of collecting. Did it ever cause any issue?

My understanding is that we need to fix underlying bugs in case order matters in some cases but generally list sorting each time looks like an overkill and can slowdown significantly as well.

Is there a way to put files and directories in pre-determined order into filesystem maybe?

[ihor-sviziev]

@ihor-sviziev I see, just some class list in order of collecting. Did it ever cause any issue?

My understanding is that we need to fix underlying bugs in case order matters in some cases but generally list sorting each time looks like an overkill and can slowdown significantly as well.

Is there a way to put files and directories in pre-determined order into filesystem maybe?

I’m not sure if it ever created issues there, but you even can’t test it because result even on single machine might be different now, you can’t say that this version is correct and this is not.
Look at my PR - it adds one very simple sorting when we already get all paths to the modules. During my tests it’s not slowed down DI compile step, so it’s not a big deal

There actually similar issues with static content deploy - you’re getting different content of files js-translation.json and different content of js bundle files, but I still didn’t get to it.

[orlangur]

Sorry, my bad, if we just sort paths and not full list of files sorting may be negligible. Will double check.

I’m not sure if it ever created issues there, but you even can’t test it because result even on single machine might be different now, you can’t say that this version is correct and this is not.

I meant real bugs caused by nondeterministic order of items in list.

There actually similar issues with static content deploy - you’re getting different content of files js-translation.json and different content of js bundle files, but I still didn’t get to it.

Does Symfony enforce SB with some build?

[ihor-sviziev]

I meant real bugs caused by nondeterministic order of items in list.

I didn't saw any real bugs caused by this issue. I think just because this issue is not persistent.

Does Symfony enforce SB with some build?

I reviewed all PRs linked to symfony/symfony#25958 - found only one check - https://github.com/symfony/symfony/pull/26176/files#diff-f410485b2fdabe90d3c911496fbe9ea9R64
Not sure if we can do similar check

Actually test also could be like this:
https://github.com/lstrojny/symfony-reproducible-builds/blob/master/test-reproducible-build.sh

[ihor-sviziev]

Hi @orlangur,
Any updates?

[orlangur]

@ihor-sviziev sorry for delay, will post an update this week.