Skip to content

Reenable individual auth tests #26823

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Oct 28, 2020
Merged

Reenable individual auth tests #26823

merged 5 commits into from
Oct 28, 2020

Conversation

HaoK
Copy link
Member

@HaoK HaoK commented Oct 12, 2020

Fixes #26776

@ghost ghost added the area-mvc Includes: MVC, Actions and Controllers, Localization, CORS, most templates label Oct 12, 2020
@HaoK
Copy link
Member Author

HaoK commented Oct 16, 2020

@dotnet/aspnet-blazor-eng @dougbu how do we feel about maybe moving project template tests back to azdo for now (off helix), that would let us unquarantine a lot of tests that are broken due to issues around installing our custom bits correctly on the helix machines. Once the work involving moving to a model of testing against a coherent SDK, we can switch back to helix. It doesn't seem worth trying to hack something short term if we are moving away from our current model of testing. Thoughts?

@dougbu
Copy link
Contributor

dougbu commented Oct 16, 2020

I don't think this is the right short-term solution because

  1. it's essentially an admission of failure
  2. it takes pressure off getting a real fix
  3. we need to get more happening on Helix, not less, in order to take advantage of @davidfowl's progress making blame, dumps, et cetera widely available

Besides all that, we are basically in MQ and should spend some of that period to really fix whatever Helix infrastructure problems we've got.

@pranavkm
Copy link
Contributor

@HaoK could you elaborate what it is that's problematic here and how that would be resolved in the future:

broken due to issues around installing our custom bits correctly on the helix machines

@HaoK
Copy link
Member Author

HaoK commented Oct 16, 2020

Sure, so the main issue with testing templates is that we have to basically install the correct bits onto the helix machines. This actually fits perfectly into wider long term future goal where we test against a coherent product build as opposed to whatever it is that we generally cobble together (which is especially true for our template tests).

To summarize where we are at:
The template tests were originally written with the expectation that they would run against the built outputs, like most of our tests, so they've have had a lot of years of iteration and hardening running things the 'old' way on azdo.. I did some work in the last few years to add enough tweaks for the tests to run on helix in a few limited scenarios (non arm, no selenium, always using windows build shared runtime bits, etc). But basically we need to continue to maintain and improve a fragile infrastructure to basically install enough of a subset of our bits on the helix machine that our template tests need.

If the end goal is to start just pointing our tests against a version of the SDK, that removes the need to have any kind of install our private bits step on the helix machines. The flow would just be, send the correct SDK version as part of our helix jobs that contains the coherent SDK with templates, and then run the tests. Since things already work well enough on azdo, I'm suggesting we consider abandoning the logic which tries to install template bits on the helix machines and just run the template tests on azdo (We still run the blazor templates on azdo anyways). We don't lose much of the dump/hang work since most of that is part of dotnet [vs]test anyways.

Also we will have to do work to update the template tests to work against official sdk bits anyways, irrespective of whatever short term fixes we do here.

@dougbu
Copy link
Contributor

dougbu commented Oct 16, 2020

@HaoK I agree it would be nice if we were building against entire SDKs but we're not there yet and we don't know when in the 6.0 timeframe that will come. We need to be testing thoroughly long before that.

@pranavkm I also would like to call attention to #26776 and my comment here:

the real problem is all of the template tests run against the Microsoft.AspNetCore.App assemblies included in the SDK rather than what we just built.

We have support for things like the AzDO / local template tests' TemplateTests.props for many other tests that run on Helix but nothing for the template tests ☹️

@HaoK
Copy link
Member Author

HaoK commented Oct 16, 2020

We need to be testing thoroughly long before that.

Right this actually helps my case. So right now we have degraded testing as more tests are quarantined or skipped on helix due to it just being less functional then letting them continue to run on azdo. So if our goal its just to have better coverage today, we should just run template tests on azdo. It just seems like a poor investment to enhance/build new test install logic that we know is most likely going to be obsolete and nuked in 6.0.

@dougbu
Copy link
Contributor

dougbu commented Oct 16, 2020

My point was we don't know when testing against full ADKs will even start. And, the issue is not about moving back to AzDO. If that was enough, we'd be satisfied with the template job in PR builds.

@HaoK HaoK changed the title Investigate individual auth Reenable individual auth tests Oct 28, 2020
@HaoK HaoK marked this pull request as ready for review October 28, 2020 19:08
@HaoK HaoK requested a review from a team October 28, 2020 19:08
@HaoK HaoK merged commit 23207d7 into master Oct 28, 2020
@HaoK HaoK deleted the haok/projtemp branch October 28, 2020 19:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-mvc Includes: MVC, Actions and Controllers, Localization, CORS, most templates
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[master] Get IndividualAuth template tests working on Helix
3 participants