CI: Matplotlib 3.6 broke our builds

[phofl]

The version is only pulled on arm builds now, hence only those fail

https://app.circleci.com/pipelines/github/pandas-dev/pandas/16140/workflows/031dcf5b-bf77-40f0-9126-d32a20ba8863/jobs/46971

[rhshadrach]

In #48580 I saw the same failure on other builds

https://github.com/pandas-dev/pandas/actions/runs/3071594332/jobs/4965256242

[phofl]

Yep, looks like we are pulling the new matplotlib everywhere

Opened #48601

[mroeschke]

Looks like we are successfully testing with matplotlib 3.6 now so closing