Custom log ingestion (tables, DCR destination etc)

[Ryan-Palmer]

This PR closes #1171

I've reverse-engineered the setup needed for log ingestion by setting up Serilog through the portal and inspecting the ARM.

See the full SAFE stack demo example with raw JSON resources here:
https://github.com/Ryan-Palmer/Safe-Serilog-Farmer-Demo

The changes in this PR are as follows:

• Added custom tables to logAnalytics builder. Auto append _CL to the table names (as the portal does).

• New Column type used in Table and DCR config
• Added LogAnalytics destination to the existing DCR builder.
• Added TransformKQL and OutputStream properties to the DataFlow config
• Added StreamDeclarations to the DCR config

Stream declarations: Declaration of the different types of data sent into the Log Analytics workspace. Each stream is an object whose key represents the stream name, which must begin with Custom-

• ⚠️ Silent breaking change: Modified the CustomStream printer to prepend Custom- (as the portal does) and append _CL (as the portal requires). This allows passing of the plain table name. Could make a new case CustomStreamAutoFormat or something if this is likely to be an issue? Or just rely on people finding some obscure docs or experimenting in the portal to figure it out.?

• ⚠️ Breaking change: Made OsType optional as setting it to any value causes the DCR to fail in the logging scenario (not sure about others?)

I have read the contributing guidelines and have completed the following:

• Tested my code end-to-end against a live Azure subscription.
• Updated the documentation in the docs folder for the affected changes.
• Written unit tests against the modified code that I have made.
• Updated the release notes with a new entry for this PR.
• Checked the coding standards outlined in the contributions guide and ensured my code adheres to them.

If I haven't completed any of the tasks above, I include the reasons why here:

• I'll add docs and tests and push them up shortly, I just wanted to get the PR up for discussion in case I need to make changes.
• I don't know what release number to use, it depends on when it gets merged?

Below is a minimal example configuration that includes the new features, which can be used to deploy to Azure:

open Farmer
open Farmer.Builders
open Farmer.Arm.LogAnalytics
open Farmer.Builders.AzureMonitor
open Farmer.Arm.Monitor

let loggingName = "logging-workspace"
let tableName = "loggingTable"
let serilogColumns = [
    {
        Name = "TimeGenerated"
        Type = DateTime
    }
    {
        Name = "Event"
        Type = Dynamic
    }
]
    
let tableConfig = {
    Name = ResourceName tableName
    Plan = Analytics (Some 30<Days>)
    Columns = serilogColumns
    TotalRetentionInDays = None
}


let logging = logAnalytics {
    name loggingName
    custom_tables [ tableConfig ]
}

let tableResourceId =
    tableConfig.BuildResources (logging :> IBuilder).ResourceId
    |> List.head
    |> _.ResourceId

let dce = dataCollectionEndpoint {
    name "dce-logging"
}

let stream = CustomStream tableName

let dcr = dataCollectionRule {
    name "dcr-logging"
    endpoint (dce :> IBuilder).ResourceId
    depends_on [ tableResourceId ]
    stream_declarations [
        stream, serilogColumns
    ]
    destinations [
        LogAnalytics [
            {
                WorkspaceResourceId = (logging :> IBuilder).ResourceId
                Name = ResourceName loggingName
            }
        ]
    ]
    data_flows [
        {
            Destinations = [ loggingName ]
            Streams = [ stream ]
            TransformKQL = Some "source"
            OutputStream = Some stream
        }
    ]
}

let deployment = arm {
    location Location.UKSouth
    add_resource logging
    add_resource dce
    add_resource dcr
}

deployment
|> Deploy.execute "test-dcr" Deploy.NoParameters
|> ignore