Intel Mac OS Monterey, unsupported tapi file type '!tapi-tbd' in YAML file

[wpettine]

Description of your problem

I've been trying to get pymc3 working with the most basic examples. It keeps throwing an error, seemingly related to theano and the C compiler. Following the suggestions of various suggestions posted on the interweb, I've uninstalled and reinstalled Mac's command line tools (it wouldn't let me to downgrade to v11.5), as well as anaconda. Still getting the error though.

For my environment, I've just installed the minimal components using the command:

conda create -c conda-forge -n py39mc python=3.9 pymc3 theano-pymc mkl mkl-service jupyterlab matplotlib numpy
For the heck of it, I tried using python 3.7 but it made no difference.

Please provide a minimal, self-contained, and reproducible example.

All I've been trying to do is run the initial code in the tutorial: [https://docs.pymc.io/en/v3/pymc-examples/examples/diagnostics_and_criticism/model_comparison.html]

import arviz as az
import matplotlib.pyplot as plt
import numpy as np
import pymc3 as pm

J = 8
y = np.array([28, 8, -3, 7, -1, 1, 18, 12])
sigma = np.array([15, 10, 16, 11, 9, 11, 10, 18])

with pm.Model() as pooled:
    mu = pm.Normal("mu", 0, sigma=1e6)

    obs = pm.Normal("obs", mu, sigma=sigma, observed=y)

    trace_p = pm.sample(2000, return_inferencedata=True)

Please provide the full traceback.

Complete error traceback

---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
/var/folders/pg/y85sv5nd2r547mg2rlh7j9tr0000gn/T/ipykernel_43387/2591169388.py in <module>
      9 
     10 with pm.Model() as pooled:
---> 11     mu = pm.Normal("mu", 0, sigma=1e6)
     12 
     13     obs = pm.Normal("obs", mu, sigma=sigma, observed=y)

~/opt/anaconda3/envs/py39mc3/lib/python3.9/site-packages/pymc3/distributions/distribution.py in __new__(cls, name, *args, **kwargs)
    119             dist = cls.dist(*args, **kwargs, shape=shape)
    120         else:
--> 121             dist = cls.dist(*args, **kwargs)
    122         return model.Var(name, dist, data, total_size, dims=dims)
    123 

~/opt/anaconda3/envs/py39mc3/lib/python3.9/site-packages/pymc3/distributions/distribution.py in dist(cls, *args, **kwargs)
    128     def dist(cls, *args, **kwargs):
    129         dist = object.__new__(cls)
--> 130         dist.__init__(*args, **kwargs)
    131         return dist
    132 

~/opt/anaconda3/envs/py39mc3/lib/python3.9/site-packages/pymc3/distributions/continuous.py in __init__(self, mu, sigma, tau, sd, **kwargs)
    486 
    487         self.mean = self.median = self.mode = self.mu = mu = tt.as_tensor_variable(floatX(mu))
--> 488         self.variance = 1.0 / self.tau
    489 
    490         assert_negative_support(sigma, "sigma", "Normal")

~/opt/anaconda3/envs/py39mc3/lib/python3.9/site-packages/theano/tensor/var.py in __rtruediv__(self, other)
    174 
    175     def __rtruediv__(self, other):
--> 176         return theano.tensor.basic.true_div(other, self)
    177 
    178     def __rfloordiv__(self, other):

~/opt/anaconda3/envs/py39mc3/lib/python3.9/site-packages/theano/graph/op.py in __call__(self, *inputs, **kwargs)
    251 
    252         if config.compute_test_value != "off":
--> 253             compute_test_value(node)
    254 
    255         if self.default_output is not None:

~/opt/anaconda3/envs/py39mc3/lib/python3.9/site-packages/theano/graph/op.py in compute_test_value(node)
    124 
    125     # Create a thunk that performs the computation
--> 126     thunk = node.op.make_thunk(node, storage_map, compute_map, no_recycling=[])
    127     thunk.inputs = [storage_map[v] for v in node.inputs]
    128     thunk.outputs = [storage_map[v] for v in node.outputs]

~/opt/anaconda3/envs/py39mc3/lib/python3.9/site-packages/theano/graph/op.py in make_thunk(self, node, storage_map, compute_map, no_recycling, impl)
    632             )
    633             try:
--> 634                 return self.make_c_thunk(node, storage_map, compute_map, no_recycling)
    635             except (NotImplementedError, MethodNotDefined):
    636                 # We requested the c code, so don't catch the error.

~/opt/anaconda3/envs/py39mc3/lib/python3.9/site-packages/theano/graph/op.py in make_c_thunk(self, node, storage_map, compute_map, no_recycling)
    598                 print(f"Disabling C code for {self} due to unsupported float16")
    599                 raise NotImplementedError("float16")
--> 600         outputs = cl.make_thunk(
    601             input_storage=node_input_storage, output_storage=node_output_storage
    602         )

~/opt/anaconda3/envs/py39mc3/lib/python3.9/site-packages/theano/link/c/basic.py in make_thunk(self, input_storage, output_storage, storage_map)
   1201         """
   1202         init_tasks, tasks = self.get_init_tasks()
-> 1203         cthunk, module, in_storage, out_storage, error_storage = self.__compile__(
   1204             input_storage, output_storage, storage_map
   1205         )

~/opt/anaconda3/envs/py39mc3/lib/python3.9/site-packages/theano/link/c/basic.py in __compile__(self, input_storage, output_storage, storage_map)
   1136         input_storage = tuple(input_storage)
   1137         output_storage = tuple(output_storage)
-> 1138         thunk, module = self.cthunk_factory(
   1139             error_storage,
   1140             input_storage,

~/opt/anaconda3/envs/py39mc3/lib/python3.9/site-packages/theano/link/c/basic.py in cthunk_factory(self, error_storage, in_storage, out_storage, storage_map)
   1632             for node in self.node_order:
   1633                 node.op.prepare_node(node, storage_map, None, "c")
-> 1634             module = get_module_cache().module_from_key(key=key, lnk=self)
   1635 
   1636         vars = self.inputs + self.outputs + self.orphans

~/opt/anaconda3/envs/py39mc3/lib/python3.9/site-packages/theano/link/c/cmodule.py in module_from_key(self, key, lnk)
   1189             try:
   1190                 location = dlimport_workdir(self.dirname)
-> 1191                 module = lnk.compile_cmodule(location)
   1192                 name = module.__file__
   1193                 assert name.startswith(location)

~/opt/anaconda3/envs/py39mc3/lib/python3.9/site-packages/theano/link/c/basic.py in compile_cmodule(self, location)
   1541             try:
   1542                 _logger.debug(f"LOCATION {location}")
-> 1543                 module = c_compiler.compile_str(
   1544                     module_name=mod.code_hash,
   1545                     src_code=src_code,

~/opt/anaconda3/envs/py39mc3/lib/python3.9/site-packages/theano/link/c/cmodule.py in compile_str(module_name, src_code, location, include_dirs, lib_dirs, libs, preargs, py_module, hide_symbols)
   2544             # difficult to read.
   2545             compile_stderr = compile_stderr.replace("\n", ". ")
-> 2546             raise Exception(
   2547                 f"Compilation failed (return status={status}): {compile_stderr}"
   2548             )

Exception: ("Compilation failed (return status=1): ld: unsupported tapi file type '!tapi-tbd' in YAML file '/Library/Developer/CommandLineTools/SDKs/MacOSX12.1.sdk/usr/lib/libSystem.tbd' for architecture x86_64. clang-12: error: linker command failed with exit code 1 (use -v to see invocation). ", 'FunctionGraph(Elemwise{true_div,no_inplace}(TensorConstant{1.0}, TensorConstant{1e-12}))')

Versions and main components

• PyMC/PyMC3 Version: 3.11.4
• Aesara/Theano Version: 1.1.2
• Python Version: 3.9
• Operating system: Intel MacOS Monterey 12.2.1
• How did you install PyMC/PyMC3: conda

[twiecki]

Odd error, it could disappear when you switch to pymc v4.

[arjunc12]

I got the exact same error. PyMC3 was working for me a month ago and then it just stopped.

Was v4 developed on/for the M4 macs with monterrey?

Alternately, is there any way to check that my gcc/clang version are correct and functional?

[twiecki]

@arjunc12 did you install from the conda-forge channel? I run it without problems on an M1.

[twiecki]

And intel MKL shouldn't be compatible here, right?

[arjunc12]

@twiecki Here are my exact specifications:

• Macbook pro 16 inch (2021)
• Chip: Apple M1 Pro
• Operating system: MacOS Monterrey version 12.2.1

I have xcode version 13.3 installed

Here are my gcc/clang specs:

Apple clang version 13.1.6 (clang-1316.0.21.2)
Target: arm64-apple-darwin21.3.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

I am using conda version 4.11.0

I created the environment using the following command:

conda create -c conda-forge -n pymc3_env python=3.9 pymc3

I also installed jupyterlab into this environment

conda activate pymc3_env
conda install jupyterlab

After doing all of this, here are the specifications for my environment:

• Python version:

Python 3.9.10 | packaged by conda-forge | (main, Feb 1 2022, 21:27:48)
[Clang 11.1.0 ] on darwin

• PyMC3 version: 3.11.4
• Theano version: 1.1.2

I created a jupyter kernel associated with this environment as described here

I started my jupyter notebook. I ran the following cell

import pymc3 as pm

I get the following errors:

WARNING (theano.link.c.cmodule): install mkl with `conda install mkl-service`: No module named 'mkl'
WARNING (theano.tensor.blas): Using NumPy C-API based implementation for BLAS functions.

Next, I run the following code:
basic_model = pm.Model()

No issues there.

Now I run the following code, which produces the gross error message:

with basic_model:
    # Priors for unknown model parameters
    beta1 = pm.Normal("beta1", mu=4, sigma=1)
    beta2 = pm.Normal("beta2", mu=9.8, sigma=1)
    sigma = pm.HalfNormal("sigma", sigma=1)

I will post updates about whether installing mkl-service helped, and whether v4 helped. But please let me know if you notice anything that you think might help!

[twiecki]

Oh I see, you haven't tried v4 yet, please do. And you can't use mkl with M1.

[arjunc12]

Ok so I did install PyMC3 v4 according to the instructions here https://github.com/pymc-devs/pymc/wiki/Installation-Guide-(MacOS)

The good news is that I am now able to get my basic model working. I am back to being able to compute the MAP estimate and run MCMC

The bad news is that I can't use arviz anymore!

I tried to run the following code:

import arviz as az
az.style.use("arviz-darkgrid")
with basic_model:
    az.plot_trace(trace)

I get the following error trace:

> ---------------------------------------------------------------------------
> ModuleNotFoundError                       Traceback (most recent call last)
> Input In [13], in <cell line: 1>()
>       1 with basic_model:
> ----> 2     az.plot_trace(trace)
> 
> File ~/opt/anaconda3/envs/pymc-dev-py39/lib/python3.9/site-packages/arviz/plots/traceplot.py:171, in plot_trace(data, var_names, filter_vars, transform, coords, divergences, kind, figsize, rug, lines, circ_var_names, circ_var_units, compact, compact_prop, combined, chain_prop, legend, plot_kwargs, fill_kwargs, rug_kwargs, hist_kwargs, trace_kwargs, rank_kwargs, labeller, axes, backend, backend_config, backend_kwargs, show)
>     169 if divergences:
>     170     try:
> --> 171         divergence_data = convert_to_dataset(data, group="sample_stats").diverging
>     172     except (ValueError, AttributeError):  # No sample_stats, or no `.diverging`
>     173         divergences = None
> 
> File ~/opt/anaconda3/envs/pymc-dev-py39/lib/python3.9/site-packages/arviz/data/converters.py:179, in convert_to_dataset(obj, group, coords, dims)
>     140 def convert_to_dataset(obj, *, group="posterior", coords=None, dims=None):
>     141     """Convert a supported object to an xarray dataset.
>     142 
>     143     This function is idempotent, in that it will return xarray.Dataset functions
>    (...)
>     177     xarray.Dataset
>     178     """
> --> 179     inference_data = convert_to_inference_data(obj, group=group, coords=coords, dims=dims)
>     180     dataset = getattr(inference_data, group, None)
>     181     if dataset is None:
> 
> File ~/opt/anaconda3/envs/pymc-dev-py39/lib/python3.9/site-packages/arviz/data/converters.py:91, in convert_to_inference_data(obj, group, coords, dims, **kwargs)
>      89         return from_pystan(**kwargs)
>      90 elif obj.__class__.__name__ == "MultiTrace":  # ugly, but doesn't make PyMC3 a requirement
> ---> 91     return from_pymc3(trace=kwargs.pop(group), **kwargs)
>      92 elif obj.__class__.__name__ == "EnsembleSampler":  # ugly, but doesn't make emcee a requirement
>      93     return from_emcee(sampler=kwargs.pop(group), **kwargs)
> 
> File ~/opt/anaconda3/envs/pymc-dev-py39/lib/python3.9/site-packages/arviz/data/io_pymc3_3x.py:580, in from_pymc3(trace, prior, posterior_predictive, log_likelihood, coords, dims, model, save_warmup, density_dist_obs)
>     528 def from_pymc3(
>     529     trace=None,
>     530     *,
>    (...)
>     538     density_dist_obs=True,
>     539 ):
>     540     """Convert pymc3 data into an InferenceData object.
>     541 
>     542     All three of them are optional arguments, but at least one of ``trace``,
>    (...)
>     578     InferenceData
>     579     """
> --> 580     return PyMC3Converter(
>     581         trace=trace,
>     582         prior=prior,
>     583         posterior_predictive=posterior_predictive,
>     584         log_likelihood=log_likelihood,
>     585         coords=coords,
>     586         dims=dims,
>     587         model=model,
>     588         save_warmup=save_warmup,
>     589         density_dist_obs=density_dist_obs,
>     590     ).to_inference_data()
> 
> File ~/opt/anaconda3/envs/pymc-dev-py39/lib/python3.9/site-packages/arviz/data/io_pymc3_3x.py:75, in PyMC3Converter.__init__(self, trace, prior, posterior_predictive, log_likelihood, predictions, coords, dims, model, save_warmup, density_dist_obs)
>      61 def __init__(
>      62     self,
>      63     *,
>    (...)
>      73     density_dist_obs: bool = True,
>      74 ):
> ---> 75     import pymc3
>      77     try:
>      78         import aesara  # pylint: disable=redefined-outer-name
> 
> ModuleNotFoundError: No module named 'pymc3'

Is this just because arviz is still being updated?

[arjunc12]

Also the results that I am getting are not the same as the results that I used to get on the same data. But I can go to Friday's office hours to get help with that!

[arjunc12]

Also thank you for all of your help :)

[twiecki]

I think your arviz is out-of-date.

[arjunc12]

I have arviz version 11.4, which is the latest version on conda-forge.

Note that I have the arviz that came from following the instructions to install v4 from the repository itself, so if that arviz is out of date then the v4 arviz is out of date

[arjunc12]

Ah, figured out the issue.

When I computed the MCMC, I was using "return_inferenced_data=False", which is incompatible with the most up to date version of arviz.

No errors anymore :)

[drbenvincent]

No errors anymore :)

Think we can close this issue?