Description
It would be really useful if the SDK could surface the strongly-typed RequestMetrics instance to the logging subsystem (and, actually, any other telemetry) - or if there were a way to plug in an adapter to the SDK's logging system. I happen to use NLog for logging, and I emit logs in a JSON format optimised for simplicity of harvest (via nxlog) to ingest into Elasticsearch by way of LogStash.
In my use-case, this means the application logs one JSON event per line. The logging library adds JSON attributes like timestamp, process id, thread id, and so forth, via configuration.
I think because of the AWS SDK's assumption that it's in control of formatting the string that ends up in log files as the message token (eg {timestamp}|{processid}|{message} as a contrived human-readable example), that means users that want control over the format of their log lines (as opposed to the message within their log lines) have to eat another deserialisation/enrich/serialise cycle, or do something a little intrusive to the SDK.
Since the SDK is also responsible for retrying requests internally (I don't think it's possible to control that behaviour), it also emits logs (via the standard formatter) that are still hard to parse because in the event of retries, things turn into arrays (rather than are emitted as arrays with single items).
Would it be possible to take the idea of a custom log message formatter (introduced in #157), and re-apply it so that the log-framework itself can be pluggable? That would enable both the specific thing I'd like to do, and also other users to direct logs to the framework of their choice.
I was thinking about something like
<aws ...>
<logging logTo="custom,KeepTheArrayButIWouldntNeedIt" customLogAdapter="type name, assembly"/>
</aws>and the adapter's signature being something like:
public interface ILogEvents {
void Log(LogEvent @event); // LogEvent is assumed to have log-level as a property
}
public abstract class AwsSdkLogger : ILogEvents {
public virtual void Log(LogEvent @event) { // allow override if the adapter wants to completely differ, and write structured logs
var line = MapEventToLogLine(@event);
WriteLogLine(line);
}
protected virtual string MapEventToLogLine(LogEvent @event) {
//return the string as the current implementation does
//this is where a user could override if they wanted a different string in their message token in their log lines - eg, remove the newlines that the SDK emits
}
protected virtual void WriteLogLine(string line) {
// write the log to whatever sink
// this is where a user could override to plug their logger in directly, rather than going via System.Diagnostics
}
}At that point, the logging concerns of the SDK are logically separate and really easy to customise to fit gracefully into the host application's existing setup.
For example, NLog sketch
public class NLogAWSSDKLogAdapter : AwsSdkLogger {
public class NLogAWSSDKLogAdapter() {
_logger = LogManager.GetLogger("Amazon");
}
public override void Log(LogEvent @event) {
var nlEvent = MapToNLog(@event);
_logger.Log(nlEvent);
}
public NLog.LogEvent(LogEvent @event) {
var e = new NLog.LogEvent();
e.Properties["Attempts"] = PullOutRequestAttemptsData(@event);
e.Properties["aws_ServiceName"] = @event.ServiceName;
/// etc, on the basis that NLog itself will now be in charge of rendering a string for the event with its own rich layoutrenderers and layouts per target, so different log sinks can receive different messages as appropriate (eg, hipchat gets a short note and a link to a kibana dashboard, whereas logstash gets a fully structured JSON blob that it will pull apart into Elasticsearch)
return e;
}
}It's crucial, really, for the value of this feature, that the change make available the log-event instances (IRequestMetrics and any others) to the adapter, and delegate serialisation / formatting entirely. If the consumer needs to deal with ETL, we won't have improved the current implementation.
NLog and other frameworks also have memory log-sinks, which is invaluable for testing - one can both read logs within test automation, but also assert against their content without needing to wrap standard out (for example).