PyPlot interferes with LinearAlgebra in a weird way

[atbug]

Let me start with how to reproduce:

julia> using LinearAlgebra, PyPlot

julia> xlim([-1, 1]) # a window will pop up, leave the window open.
(-1, 1)

julia> H = rand(46, 46)+im*rand(46, 46);

julia> values, vectors = eigen(Hermitian(H))
Eigen{Complex{Float64},Float64,Array{Complex{Float64},2},Array{Float64,1}}
values:
0-element Array{Float64,1} # should have been a 46 element array
vectors:
46×0 Array{Complex{Float64},2}

julia> versioninfo() # official binary
Julia Version 1.4.0
Commit b8e9a9ecc6 (2020-03-21 16:36 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i5-8265U CPU @ 1.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-8.0.1 (ORCJIT, skylake)

(@v1.4) pkg> st
Status `C:\Users\wangc\.julia\environments\v1.4\Project.toml`
  [c52e3926] Atom v0.12.10
  [6e4b80f9] BenchmarkTools v0.5.0
  [861a8166] Combinatorics v1.0.0
  [27194a60] Hop v0.1.0 [`D:/Workspace/Hop.jl`]
  [7073ff75] IJulia v1.21.1
  [6a3955dd] ImageFiltering v0.6.11
  [e5e0dc1b] Juno v0.8.1
  [5b5b96b6] MoireRelaxation v0.1.0 [`D:/Workspace/MoireRelaxation.jl`]
  [429524aa] Optim v0.20.6
  [438e738f] PyCall v1.91.4
  [d330b81b] PyPlot v2.9.0

Several observations:

• Only the first run of values, vectors = eigen(Hermitian(H)) will give the wrong result. If the statement is run a second time, the eigenvalues and eigenvectors are correct.
• I am not sure this is a Windows issue since I don't have linux machines with gui. To reproduce, the gui is essential: if PyPlot is started with Agg backend, the eigenvalues are correct.
• The number 46 is not very special and 24, 12 would also reproduce this. However, if the number is 4. I cannot reproduce this on my machine.
• If I collect all the statements in a file run.jl and do include(run.jl) in the REPL, I cannot reproduce this.

Python package list:

(base) C:\Users\wangc>conda list
# packages in environment at C:\Users\wangc\.julia\conda\3:
#
# Name                    Version                   Build  Channel
anaconda-client           1.7.2                    py37_0
anaconda-navigator        1.9.12                   py37_0
apscheduler               3.6.3                    py37_0
ase                       3.19.1                     py_0    conda-forge
asn1crypto                1.3.0                    py37_0
attrs                     19.3.0                     py_0
backcall                  0.1.0                    py37_0
blas                      1.0                         mkl
bleach                    3.1.0                    py37_0
bzip2                     1.0.8                he774522_0
ca-certificates           2020.1.1                      0
certifi                   2020.4.5.1               py37_0
cffi                      1.14.0           py37h7a1dbc1_0
cftime                    1.1.1.1          py37h2a96729_0
chardet                   3.0.4                 py37_1003
click                     7.1.1                      py_0
cloudpickle               1.3.0                      py_0
clyent                    1.2.2                    py37_1
colorama                  0.4.3                      py_0
conda                     4.8.3                    py37_0
conda-package-handling    1.6.0            py37h62dcd97_0
console_shortcut          0.1.1                         4
cryptography              2.8              py37h7a1dbc1_0
curl                      7.69.1               h2a8f88b_0
cycler                    0.10.0                   py37_0
cytoolz                   0.10.1           py37he774522_0
dask-core                 2.14.0                     py_0
decorator                 4.4.2                      py_0
defusedxml                0.6.0                      py_0
entrypoints               0.3                      py37_0
expat                     2.2.5                he025d50_0
fastcache                 1.1.0            py37he774522_0
flask                     1.1.1                      py_1
freetype                  2.9.1                ha9979f8_1
future                    0.18.2                   py37_0
geos                      3.8.0                h33f27b4_0
hdf4                      4.2.13               h712560f_2
hdf5                      1.10.4               h7ebc959_0
icc_rt                    2019.0.0             h0cc432a_1
icu                       58.2                 ha66f8fd_1
idna                      2.9                        py_1
imageio                   2.8.0                      py_0
importlib_metadata        1.5.0                    py37_0
intel-openmp              2020.0                      166
ipykernel                 5.1.4            py37h39e3cac_0
ipython                   7.13.0           py37h5ca1d4c_0
ipython_genutils          0.2.0                    py37_0
itsdangerous              1.1.0                    py37_0
jedi                      0.16.0                   py37_1
jinja2                    2.11.1                     py_0
jpeg                      9b                   hb83a4c4_2
json5                     0.9.4                      py_0
jsoncpp                   1.8.4                h74a9793_0
jsonschema                3.2.0                    py37_0
jupyter_client            6.1.2                      py_0
jupyter_core              4.6.3                    py37_0
jupyterlab                1.2.6              pyhf63ae98_0
jupyterlab_server         1.1.0                      py_0
kiwisolver                1.1.0            py37ha925a31_0
krb5                      1.17.1               hc04afaa_0
latexcodec                2.0.0                      py_0    conda-forge
libcurl                   7.69.1               h2a8f88b_0
libiconv                  1.15                 h1df5818_7
libnetcdf                 4.6.1                h411e497_2
libpng                    1.6.37               h2a8f88b_0
libsodium                 1.0.16               h9d3ae62_0
libssh2                   1.9.0                h7a1dbc1_1
libtiff                   4.1.0                h56a325e_0
libxml2                   2.9.9                h464c3ec_0
lz4-c                     1.8.1.2              h2fa13f4_0
m2w64-gcc-libgfortran     5.3.0                         6
m2w64-gcc-libs            5.3.0                         7
m2w64-gcc-libs-core       5.3.0                         7
m2w64-gmp                 6.1.0                         2
m2w64-libwinpthread-git   5.0.0.4634.697f757               2
markupsafe                1.1.1            py37he774522_0
matplotlib                3.1.3                    py37_0
matplotlib-base           3.1.3            py37h64f37c6_0
menuinst                  1.4.16           py37he774522_0
mistune                   0.8.4            py37he774522_0
mkl                       2020.0                      166
mkl-service               2.3.0            py37hb782905_0
mkl_fft                   1.0.15           py37h14836fe_0
mkl_random                1.1.0            py37h675688f_0
monty                     3.0.2                      py_0    conda-forge
mpmath                    1.1.0                    py37_0
msys2-conda-epoch         20160418                      1
nbconvert                 5.6.1                    py37_0
nbformat                  5.0.4                      py_0
netcdf4                   1.4.2            py37h812ae01_0
networkx                  2.4                        py_0
nodejs                    10.13.0                       0
notebook                  6.0.3                    py37_0
numpy                     1.18.1           py37h93ca92e_0
numpy-base                1.18.1           py37hc3f5095_1
olefile                   0.46                     py37_0
openssl                   1.1.1f               he774522_0
palettable                3.3.0                      py_0
pandas                    0.25.1           py37ha925a31_0
pandoc                    2.2.3.2                       0
pandocfilters             1.4.2                    py37_1
parso                     0.6.2                      py_0
pickleshare               0.7.5                    py37_0
pillow                    7.0.0            py37hcc1f983_0
pip                       20.0.2                   py37_1
plotly                    4.6.0                      py_0
powershell_shortcut       0.0.1                         3
prometheus_client         0.7.1                      py_0
prompt-toolkit            3.0.4                      py_0
prompt_toolkit            3.0.4                         0
psutil                    5.7.0            py37he774522_0
pybtex                    0.22.2           py37hc8dfbb8_1    conda-forge
pycosat                   0.6.3            py37he774522_0
pycparser                 2.20                       py_0
pydispatcher              2.0.5                    py37_1
pygments                  2.6.1                      py_0
pymatgen                  2020.4.2         py37heaa310e_0    conda-forge
pyopenssl                 19.1.0                   py37_0
pyparsing                 2.4.6                      py_0
pyqt                      5.9.2            py37h6538335_2
pyrsistent                0.16.0           py37he774522_0
pysocks                   1.7.1                    py37_0
python                    3.7.7           h60c2a47_0_cpython
python-dateutil           2.8.1                      py_0
python_abi                3.7                     1_cp37m    conda-forge
pytz                      2019.3                     py_0
pywavelets                1.1.1            py37he774522_0
pywin32                   227              py37he774522_1
pywinpty                  0.5.7                    py37_0
pyyaml                    5.3.1            py37he774522_0
pyzmq                     18.1.1           py37ha925a31_0
qt                        5.9.7            vc14h73c81de_0
qtpy                      1.9.0                      py_0
requests                  2.23.0                   py37_0
retrying                  1.3.3                    py37_2
ruamel.yaml               0.16.5           py37he774522_1
ruamel.yaml.clib          0.2.0            py37he774522_0
ruamel_yaml               0.15.87          py37he774522_0
scikit-image              0.16.2           py37h47e9c7a_0
scipy                     1.4.1            py37h9439919_0
send2trash                1.5.0                    py37_0
setuptools                46.1.3                   py37_0
shapely                   1.6.4            py37hd830579_0
sip                       4.19.8           py37h6538335_0
six                       1.14.0                   py37_0
spglib                    1.14.1           py37hbc2f12b_1    conda-forge
sqlite                    3.31.1               he774522_0
sympy                     1.5.1                    py37_0
tabulate                  0.8.3                    py37_0
tbb                       2020.0               h74a9793_0
terminado                 0.8.3                    py37_0
testpath                  0.4.4                      py_0
tk                        8.6.8                hfa6e2cd_0
toolz                     0.10.0                     py_0
tornado                   6.0.4            py37he774522_1
tqdm                      4.44.1                     py_0
traitlets                 4.3.3                    py37_0
tzlocal                   2.0.0                    py37_0
urllib3                   1.25.8                   py37_0
vc                        14.1                 h0510ff6_4
vs2015_runtime            14.16.27012          hf0eaf9b_1
vtk                       8.2.0           py37h1e53df8_200
wcwidth                   0.1.9                      py_0
webencodings              0.5.1                    py37_1
werkzeug                  1.0.0                      py_0
wheel                     0.34.2                   py37_0
win_inet_pton             1.1.0                    py37_0
wincertstore              0.2                      py37_0
winpty                    0.4.3                         4
xmltodict                 0.12.0                     py_0
xz                        5.2.4                h2fa13f4_4
yaml                      0.1.7                hc54c509_2
zeromq                    4.3.1                h33f27b4_3
zipp                      2.2.0                      py_0
zlib                      1.2.11               h62dcd97_3
zstd                      1.3.7                h508b16e_0

[stevengj]

I can't reproduce — it seems to work fine for me on MacOS with Julia 1.4. So it may be a Windows thing, e.g. some strange interference between numpy's LAPACK library and Julia's?

[atbug]

I cannot reproduce this on a windows server 2016 virtual machine. What can I do to figure out what's wrong with my machine?

[zhopan77]

I also have met similar issues on a single PC with Windows 10 64-bit, with Julia v1.4.2, PyPlot v2.9.0, Polynomials v1.1.3, and CSV v0.7.4.

In a script that I first run through some PyPlot plotting commands, Polynomials.fit() may do one of the following:

• Generate a fit with all NaN results (but a second call to fit() will produce correct result)
• LAPACKException(3) or (4).
  Not all Polynomials.fit() in the same script would fail. But once a failure happens, it is very repeatable i.e. running the same scripts multiple times would produce the same error at the same fit() call.

And sometimes CSV.read() will fail with error message like: ArgumentError: "data.csv" is not a valid file. However, this file has been loaded perfectly fine elsewhere.

If I comment out the plotting, all these issues go away. (I can still "using PyPlot" - that doesn't seem to hurt.)

See this issue I submitted to Polynomials.jl for details.

Tried pointing ENV["PYTHON"] to the MiniConda Python coming with Julia, to an ActivePython 3.7 installation, or to a standard Python 3.8 installation (which has no MKL), and rebuilding PyCall. Doesn't seem to matter for these issues.

I have another machine with the same CPU (Intel i9-9900K) and same Julia 1.4.2 and PyPlot v2.9.0, but there I installed MKL.jl so BLAS.vendor() is MKL instead of openblas. (On this machine I have to use the standard Python with no MKL to avoid conflict between Python's numpy MKL and Julia's MKL, which is a known problem.) These issues do not happen on this machine.

On older Julia (v1.4.1 and before) I have never seen this. But of course other packages may also be older at that time.

[zhopan77]

Following up on my comment above: I found the workaround for this particular PC: install MKL.jl.

By doing so I limit myself to a standard Python installation without MKL for matplotlib. However, this works around all the weird issues I reported above.

[stevengj]

If you are using MKL, there is the danger of a conflict between its BLAS library and the MKL library that is installed by many Python distributions (e.g. Anaconda) for Numpy — see JuliaPy/PyCall.jl#443.

With the default Julia OpenBLAS library, however, I didn't think there would be a symbol conflict, at least on 64-bit machines?

[atbug]

@zhopan77 Can you reproduce my original example?

BTW: I can reproduce my original issue in julia-1.5.0-rc1.

[zhopan77]

@stevengj Yes I'm aware of the MKL conflict between Julia and Python. For that purpose I installed the standard Python from https://www.python.org/ and installed only matplotlib on it, and I pointed Julia to this special installation rather than the typical Python distribution I use by default. It's wasteful but works. : -)

[zhopan77]

@atbug I already switched over to MKL so the issues won't happen anymore... Maybe I will reinstall the openblas Julia and try it some time later.

[atbug]

A list of processors that might reproduce this issue: i5-9265U; i9-9900 (from https://discourse.julialang.org/t/bug-in-linearalgebra-plots-pyplot-combo-on-intel-i9-9900/44097). @zhopan77 what's your processor?

[alecloudenback]

I don't reproduce this with the 1.5.0 release on 9900k OpenBLAS on Windows:

               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.5.0 (2020-08-01)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using LinearAlgebra, PyPlot

julia>  xlim([-1, 1]) # a window will pop up, leave the window open
(-1, 1)

julia> H = rand(46, 46)+im*rand(46, 46);

julia>  values, vectors = eigen(Hermitian(H))
Eigen{Complex{Float64},Float64,Array{Complex{Float64},2},Array{Float64,1}}
values:
46-element Array{Float64,1}:
 -9.648144895450402
 -6.403018195231846
 -5.567006204994933
 -4.651101399349197
 -4.360775788309995
 -3.924667423840675
 -3.629646918347601
 -3.4363134351487616
 -3.00654617573785
 -2.783730791156831
 -2.578633073843469
 -2.242173850537144
 -2.1137776117369507
  ⋮
  2.5661762242605484
  2.740609665523724
  2.92585837174361
  3.2168926169092042
  3.424265999066767
  3.825152916221416
  4.129331637473747
  4.434268297449822
  4.733944831449584
  5.461958227240098
  7.553837821773461
 29.425237094361222
vectors:
46×46 Array{Complex{Float64},2}:
 -0.0380281-0.166063im    -0.0560625-0.0569382im   …    0.078757+0.165303im     0.00268687-0.134841im
 -0.0381843-0.163177im    -0.0656187-0.135121im         0.147389+0.193109im     0.00127123-0.154623im
 -0.0252568-0.128033im     0.0744523-0.171277im        0.0972979+0.122269im     -0.0170815-0.124605im
  0.0182624-0.177805im      0.163453-0.153465im         0.153768+0.0704818im   -0.00642343-0.130077im
  0.0799447-0.109911im      0.170979+0.00691165im       0.226077+0.0746839im     0.0186681-0.114621im
  0.0541363-0.100239im      0.164807-0.127734im    …    0.149009+0.00977647im   -0.0232897-0.147091im
  0.0702992-0.129334im     0.0948445+0.109305im        0.0917108-0.0836994im    -0.0235026-0.158625im
   0.071336-0.133863im      0.148317+0.0471107im        0.204562-0.101519im     -0.0233808-0.1493im
   0.147327-0.0908953im     0.167219+0.0527419im         0.14576-0.0855678im     -0.016402-0.152231im
   0.121977-0.125995im    0.00612952+0.202024im        0.0749364-0.010839im     -0.0504555-0.156991im
  0.0262741-0.0453338im    0.0715243+0.155262im    …   0.0703615-0.0880875im    -0.0242665-0.14327im
    0.11135-0.0668462im    0.0694948+0.0819725im       0.0922752-0.0579878im    -0.0450097-0.135566im
    0.14415-0.0716435im    0.0426397+0.219534im        0.0467667-0.141799im     -0.0338453-0.139842im
           ⋮                                       ⋱                                      ⋮
 -0.0331857+0.156023im     0.0373864+0.0535262im      -0.0390124+0.132547im      -0.128443-0.0774424im
 -0.0405406+0.118388im      0.133691+0.00958768im  …  0.00290895+0.157096im      -0.135272-0.0750328im
 -0.0380726+0.15514im       0.161875-0.0538643im      0.00114205+0.0416856im     -0.143744-0.0614387im
 -0.0630953+0.125791im     0.0520178+0.161339im        0.0394217+0.173115im      -0.142329-0.0620126im
 -0.0890218+0.142978im     0.0878746+0.0871438im        0.141904+0.0275529im     -0.151823-0.0438958im
 -0.0815737+0.127161im    -0.0594128+0.148887im         0.155414+0.0971197im     -0.144271-0.0430016im
  -0.114734+0.15381im      0.0261279+0.167901im    …   0.0162599-0.0125471im     -0.148537-0.0403495im
  -0.126208+0.0954169im  -0.00961876+0.119196im        0.0637185+0.0440769im     -0.157727-0.0288604im
 -0.0963627+0.134969im    -0.0947695+0.124531im        0.0713737+0.0647373im     -0.144469-0.0310239im
  -0.117345+0.0680233im   -0.0851291+0.15621im          0.176956+0.0579259im     -0.156455-0.007943im
  -0.124463+0.0792475im   -0.0777825-0.0507211im        0.160452-0.0271196im     -0.130636-0.000188684im
  -0.150394-0.0im          -0.152386-0.0im         …    0.170812+0.0im           -0.138933-0.0im

julia>  versioninfo()
Julia Version 1.5.0
Commit 96786e22cc (2020-08-01 23:44 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, skylake)
Environment:
  JULIA_NUM_THREADS = 16

(@v1.5) pkg> st
Status `C:\Users\alecl\.julia\environments\v1.5\Project.toml`
  [bdd23359] ActuaryUtilities v0.3.2
  [6e4b80f9] BenchmarkTools v0.5.0
  [a93c6f00] DataFrames v0.21.5
  [31c24e10] Distributions v0.23.4
  [e30172f5] Documenter v0.24.11
  [35a29f4d] DocumenterTools v0.1.6
  [713c75ef] Franklin v0.9.1 `https://github.com/tlienart/Franklin.jl.git#master`
  [7073ff75] IJulia v1.21.2
  [c8f0d631] LifeContingencies v0.6.1 `C:\Users\alecl\.julia\dev\LifeContingencies`
  [4780e19d] MortalityTables v0.10.2 `C:\Users\alecl\.julia\dev\MortalityTables`
  [2bd173c7] NodeJS v1.1.1
  [6fe1bfb0] OffsetArrays v1.0.4
  [5fb14364] OhMyREPL v0.5.5
  [429524aa] Optim v0.21.0
  [b98c9c47] Pipe v1.3.0
  [91a5bcdd] Plots v1.4.1
  [c3e4b0f8] Pluto v0.10.10 `C:\Users\alecl\.julia\dev\Pluto`
  [d330b81b] PyPlot v2.9.0
  [295af30f] Revise v2.7.2
  [2913bbd2] StatsBase v0.32.2
  [b8865327] UnicodePlots v1.2.0
  [44d3d7a6] Weave v0.10.2
  [ade2ca70] Dates

[atbug]

I recently bought a new laptop. This issue is still reproducible on Intel Core i7-1185G7. The OS is Windows 10 20H2.

[jebej]

I can easily reproduce this with the following code, which returns all NaNs, though running qr a second time works.

using PyPlot, LinearAlgebra
fig = figure(1);
x = rand(10,10);
plot(x); xlim(-12,0);
first(qr(x))[1]

julia setup:

julia> versioninfo()
Julia Version 1.5.2
Commit 539f3ce943 (2020-09-23 23:17 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i7-8086K CPU @ 4.00GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, skylake)
Environment:
  JULIA_CC = C:\mingw-w64\bin\gcc.exe
  JULIA_EDITOR = atom.cmd -a
  JULIA_NUM_PRECOMPILE_TASKS = 4
  JULIA_NUM_THREADS = 6
  JULIA_PKG_SERVER = pkg.julialang.org

julia> LinearAlgebra.versioninfo()
BLAS: libopenblas (OpenBLAS 0.3.9  USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell MAX_THREADS=32)
LAPACK: libopenblas64_

python setup:

>>> import numpy
>>> numpy.show_config()
blas_mkl_info:
    libraries = ['mkl_rt']
    library_dirs = ['C:/Users/Jeremy/.julia/conda/3\\Library\\lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['C:/Users/Jeremy/.julia/conda/3\\Library\\include']
blas_opt_info:
    libraries = ['mkl_rt']
    library_dirs = ['C:/Users/Jeremy/.julia/conda/3\\Library\\lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['C:/Users/Jeremy/.julia/conda/3\\Library\\include']
lapack_mkl_info:
    libraries = ['mkl_rt']
    library_dirs = ['C:/Users/Jeremy/.julia/conda/3\\Library\\lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['C:/Users/Jeremy/.julia/conda/3\\Library\\include']
lapack_opt_info:
    libraries = ['mkl_rt']
    library_dirs = ['C:/Users/Jeremy/.julia/conda/3\\Library\\lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['C:/Users/Jeremy/.julia/conda/3\\Library\\include']