Ability to reload AWSOptions after ASP.NET Core Lambda Startup

[IgorPietraszko]

The Question

I am enhancing an existing ASP.NET Core Lambda solution to add multi-tenant support. Our multitenant setup would use the same codebase (front-end, ANguar, oaded from S3 buckets and back-end, .NET Lambdas) but separate database (either by schema, physical database or physical cluster - depnding on the client).

Per tenant requests would be recognized by incoming host since each client would have their own domain (or be mapped to one of our tenant-specific subdomains). Configuration for our application resides in AWS Parameter Store and would be composed of a Common section (/myapplication/Commmon) which would contain mappings between Hosts and Logical Tenant Names. In addition, there would be tenant-specific sections, for each configured tenant, that would contain tenant-specific application configuration values (/myapplication/tenantA/, /myapplication/tenantB/).

Processing piepeline in the ASP.NET Core 3.1 application would contain custom middleware which would do the initial mapping of Host to Tenant and then load the appropriate tenant specific configuration section.
I have tried it out in a local environment and al works great. Next step is to install in in AWS and make it work which I doubt should be an issue.

One problem I have noticed is that AWSOptions are loaded in the ConfigureServices section of my ASP.NET Core app while our middleware does not run until the next Configure section. AWSOptions are loaded from "the base" configuration "key (e.g. /myapplication/AWS/UserPoolClientId) and while I have tried to reload it from a tenant-specific section using this construct:

configuration.Bind($"{tenantName}:AWS", awsOptions.Value);

But it does not seem to work.

My general question is whether there is a way to reload AWSOptions to support different Congito (and other AWS service) on a per-client/tenant basis rather than relying on the AWSOptions configured at the "base" level. Alternatively, is there a way to lazily configure Congito (or other AWS Services) to delay configuration until the client is known.

Environment

• Build Version: 2.2.2
• OS Info: Compiled on Ubuntu 20.04, runtime set to linux-x64
• Build Environment: Azure pipelines (pool Ubuntu 20.04, runtime set to linux-x64)
• Targeted .NET Platform: ASP.NET Core 3.1

---

This is a ❓ general question

[ashishdhingra]

Hi @IgorPietraszko,

Good afternoon.

Thanks for posting guidance question. Could you please share the sample code snippet to investigate further? As far as I know, there isn't an ability to reload AWSOptions in Amazon.AspNetCore.Identity.Cognito package. However, you can take some guidance from Amazon.Extensions.Configuration.SystemsManager which supports configuration reload, if a change in Systems Manager config is detected.

Thanks,
Ashish

[ppittle]

Are you trying to persist AWS Service Clients between requests? It you're setting up a Multi Tenant Lambda function, where the Cognito Client needs to be configured differently based on request data (ie headers etc), why not recreate the Cognito Client at the beginning of every request?

public APIGatewayProxyResponse Get(APIGatewayProxyRequest request, ILambdaContext context)
{
     AmazonCognitoIdentityConfig config = LoadTenantSpecificCognitoConfi(request, context);
     var client = new AmazonCognitoIdentityClient(config);

     // perform Tenant specific work
}

Alternatively, you could implement your own Client Factory to externalize some of the logic outside of your Lambda class. Something like this may work:

public interface IMultiTenantCognitoClient
{
    AmazonCognitoIdentiyClient GetCognitoClient(APIGatewayProxyRequest request, ILambdaContext context)
}

public class MultiTenantCognitoClient : IMultiTenantCognitoClient
{
     // optional
     private readonly AWSOptions _options;

     public MultiTenantCognitoClient(AWSOptions) { _options = options; }

     public AmazonCognitoIdentiyClient GetCognitoClient(APIGatewayProxyRequest request, ILambdaContext context)
     {
          // examine _options, request, and context to build a tenant specific AmazonCognitoIdentityConfig 
          var config = ...

         return new AmazonCognitoIdentityClient(config);
     }
}

You can then setup the binding for IMultiTenantCognitoClient and then construct inject a IMultiTenantCognitoClient into your Lambda function.

[IgorPietraszko]

I am using the LambdaEntryPoint class inheriting from APIGatewayProxyFunction as the entry point. This is what runs first and this is where I set up SystemsManager to access the config from two place: 1) /myapplication/Common and 2) /myapplication/<environment_name>. <environment_name> is set up as an Environment Variable on the lambda which allows SystemsManager to support multiple instances of the same Lambda as long as each is configured with a different <environment_name>.

public class LambdaEntryPoint : Amazon.Lambda.AspNetCoreServer.APIGatewayProxyFunction
{
    protected override void Init(IWebHostBuilder builder)
    {
        builder
            .ConfigureLogging(logging => logging.AddLambdaLogger(loggerOptions))
            .ConfigureAppConfiguration((context, config) =>
            {
                var env = context.HostingEnvironment;

                config.AddSystemsManager($"/myapplication/common");
                config.AddSystemsManager($"/myapplication/{env.EnvironmentName}", 
                    optional: false, 
                    reloadAfter: TimeSpan.FromMinutes(5));
            })
            .UseStartup<Startup>();
    }
    protected override void Init(IHostBuilder builder)
    {
    }
}

Next, Startup() function runs which first configures services in ConfigureServices() and then configures the application in Configure(). AWS specific services like Cognito are added in the ConfigureServices() method.

I make a decision about calling tenants based on the incoming Request's Host which is not resolved until the Configure() method. ConfigureServices() method sets up an empty TenantOption class which is then populated by Middleware invoked from Configure() method based on the Request's Host and mapped to an appropriate section in the SystemsManager (e.g. /myapplication/<environment_name/tenantA). Here are the relevant code pieces:

public class Startup
{
    public Startup(IConfiguration configuration, IWebHostEnvironment environment) 
        : base(configuration, environment) { }

    public void ConfigureServices(IServiceCollection services)
    {
        // Set up Configuration with an empty TenantOptions, to be populated in ResolveMutitenantMiddleware
        services.Configure<TenantOptions>(t => new TenantOptions());

        var awsOption = Configuration.GetAWSOptions("AWS");

        services.AddDefaultAWSOptions(awsOption);
        services.AddAWSService<IAmazonSimpleEmailService>();

        services.AddCognitoIdentity();
        services.AddTransient<CognitoSignInManager<CognitoUser>>();
        services.AddTransient<CognitoUserManager<CognitoUser>>();
    }

    public void Configure(IApplicationBuilder app, IWebHostEnvironment env,
        ApplicationDbContext dbContext, ILogger<Startup> logger, IErrorReporter errorReporter)
    {
        // To run in multitenant setup, we need to extract the host
        app.UseMultitenantSetup();
    }
}

Below code shows how I map configuration sections to TenantOptions based on the incoming Request:

public class ResolveMutitenantMiddleware
{
    private readonly RequestDelegate _next;

    public ResolveMutitenantMiddleware(RequestDelegate next)
    {
        _next = next;
    }

    public async Task InvokeAsync(
        HttpContext context, 
        IConfiguration configuration, 
        IOptions<TenantOptions> tenantOptions,
        IOptions<AWSOptions> awsOptions)
    {
        // get the host from the request
        var host = context.Request.Host;

        // extract host to tenant mapping from config
        var commonConfigSection = configuration.GetSection("Common");
        // grab tenant name based on the host
        var tenantName = commonConfigSection.GetValue<string>(host.Value);
        // bind TenantOptions to tenant specific config section
        configuration.Bind(tenantName, tenantOptions.Value);
        // re-bind AWSOptions to tenant specific ones - does not work!
        configuration.Bind($"{tenantName}:AWS", awsOptions.Value);

        await _next(context);
    }
}

So rest of my code relying on TenantOptions works fine since configuration values in TenantOptions are not used until needed later in the request pipeline where I can inject them. However it seems that AWS services configured in the ConfigureServices() method cannot be re-jigged to load their configuration from a section other than /myapplication/<environment_name>/AWS (e.g. /myapplication/<environment_name>/tenantA/AWS) nor can I repopulate AWSOptions and "reset" all AWS Service to re-configure themselves based on the new options.

Hope this sheds more light on my dilemma.

[ppittle]

Your question makes a lot more sense now, thanks for the details; and what an intersting problem.

You are correct that once a AWS Service is created it's not possible to influence the Service by mutating theAWSOptions. The ClientFactory converts AWSOptions to a Service specific Config class as part of constructing the Service and doesn't keep any reference to AWSOptions.

Instead, we can borrow from aws/aws-sdk-net#1957 and late build AWSOptions via a Factory AND then control the factory via Middleware.

I've created a small proof-of-concept that should have everything you need.

Create a helper class:

/// <summary>
/// DI Helping that allows Middleware to set a function for late binding <see cref="AWSOptions"/>.
/// NOTE:  Binding for <see cref="IAWSOptionsFactory"/> as well as any services that want to consume it
/// need to use <see cref="ServiceLifetime.Scoped"/>
/// </summary>
public interface IAWSOptionsFactory
{
    Func<IServiceProvider, AWSOptions> AWSOptionsBuilder { get; set; }
}

public class AWSOptionsFactory : IAWSOptionsFactory
{
    public Func<IServiceProvider, AWSOptions> AWSOptionsBuilder { get; set; }
}

Then in Startup

public class Startup
{
    public Startup(IConfiguration configuration)
    {
        Configuration = configuration;
    }

    public static IConfiguration Configuration { get; private set; }

    // This method gets called by the runtime. Use this method to add services to the container
    public void ConfigureServices(IServiceCollection services)
    {
        // note: it is not necessary to use AddDefaultAWSOptions().
    
        // note: AWSOptionsFactory.AWSOptionsBuilder func will be populated in middleware
        services.AddScoped<IAWSOptionsFactory, AWSOptionsFactory>();
        services.Add(new ServiceDescriptor(typeof(AWSOptions), sp => sp.GetService<IAWSOptionsFactory>().AWSOptionsBuilder(sp), ServiceLifetime.Scoped));
        services.AddAWSService<IAmazonSimpleEmailServiceV2>(lifetime: ServiceLifetime.Scoped);
		// can register multiple Services that will all consume AWSOptions from the IAWSOptionsFactory
        services.AddAWSService<IAmazonAthena>(lifetime: ServiceLifetime.Scoped);

        services.AddControllers();
    }
    
    public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
    {
        // boiler plate config removed from example for brevity.
        
        app.UseMiddleware<LateBindingAWSOptionsMiddleware>();

        app.UseEndpoints(endpoints =>
        {
            endpoints.MapControllers();
        });
    }

Then in the middleware:

public class LateBindingAWSOptionsMiddleware
{
    private readonly RequestDelegate _next;

    public LateBindingAWSOptionsMiddleware(RequestDelegate next)
    {
        _next = next;
    }

    /// <remarks>
    /// If you want to consume a global AWSOptions add a IOptions<AWSOptions> method parameter AND
    /// call servies.AddDefaultAWSOptions() in the Startup.ConfigureServices method.
    /// </remarks>
    public async Task InvokeAsync(
        HttpContext context,
        IConfiguration configuration,
        IAWSOptionsFactory optionsFactory)
    {
        
        // create AWS Options specific to this request:
        
        optionsFactory.AWSOptionsBuilder = sp =>
        {
            var awsOptions = new AWSOptions();

            // SAMPLE: to prove we can configure AWSOptions based on HttpContext,
            // get the region endpoint from the query string
            if (context.Request.Query.TryGetValue("regionEndpoint", out var regionEndpoint))
            {
                awsOptions.Region = RegionEndpoint.GetBySystemName(regionEndpoint);
            }

            return awsOptions;
        };

        await _next(context);
    }
}

Finally, there is no changes in the Controller, it can constructor inject Service Clients as normal:

[Route("api/[controller]")]
public class ValuesController : ControllerBase
{
    private readonly IAmazonSimpleEmailServiceV2 _amazonSimpleEmailService;

    public ValuesController(IAmazonSimpleEmailServiceV2 amazonSimpleEmailService)
    {
        _amazonSimpleEmailService = amazonSimpleEmailService;
    }

    // GET api/values
    [HttpGet]
    public IEnumerable<string> Get()
    {
        return new string[]
        {
            _amazonSimpleEmailService.Config.RegionEndpoint.DisplayName
        };
    }
}

To see this working:

Relative Url	Result
/api/Values?regionEndpoint="us-east-1"	["US East (N. Virginia)"]
/api/Values?regionEndpoint="us-east-2"	["US East (Ohio)"]
/api/values?regionEndpoint=eu-west-1	["Europe (Ireland)"]

[ppittle]

Interested to hear if this works for your use case or if you have any follow up questions.

[IgorPietraszko]

Looks very promising...trying it out...

[IgorPietraszko]

Getting a NullReferenceException on the highlighted line - not sure why...

            //var awsOption = Configuration.GetAWSOptions("AWS");

            //services.AddDefaultAWSOptions(awsOption);

            // note: AWSOptionsFactory.AWSOptionsBuilder func will be populated in middleware
            services.AddScoped<IAWSOptionsFactory, AWSOptionsFactory>();
            -> services.Add(new ServiceDescriptor(typeof(AWSOptions), sp => sp.GetService<IAWSOptionsFactory>().AWSOptionsBuilder(sp), ServiceLifetime.Scoped));

            services.AddAWSService<IAmazonSimpleEmailService>();

            services.AddCognitoIdentity();
            services.AddTransient<CognitoSignInManager<CognitoUser>>();
            services.AddTransient<CognitoUserManager<CognitoUser>>();

I am not familiar with ServiceDescriptor() - may need to read up on that. But the concept of a delayed provision of AWSOptions looks promising.

[ppittle]

This is just a guess without your full codebase, but I'm guessing IAWSOptionsFactory.AWSOptionsBuilder is what's null, so trying to invoke it (as it's a Func<>) will result in a Null Reference Exception.

In my example, it's the Middleware that populates IAWSOptionsFactory.AWSOptionsBuilder.

What this means is, if the Service Collection needs to construct an AWSOptions in the current Scope before the middleware has populated IAWSOptionsFactory.AWSOptionsBuilder, then you're going to get an Exception.

You should have a couple options depending on how you're initializing your application and what is trying to take a dependency on AWSOptions. You might be able to use serivces.AddDefaultAWSOptions() to define a Singleton scoped AWSOptions. Or you might need to create a custom Scope just for AWS Services created after your multi tenant middlware: https://docs.microsoft.com/en-us/dotnet/core/extensions/dependency-injection#scope-scenarios