Including (parts of) another YAML file

[mikhail-barg]

I'd like to re-open the problem raised here: #368

The proposed solution with custom tags does basically work, but it has the problem explained by @rcdailey: the INodeDeserializer.Deserialize method should return already parsed/converted type, but it has no info on the type expected by the model.

I was able to implement some basic include tag deserializer that works in general like this:

      location: http://some.example/api
      auth:
        user: !include secret.yaml#auth.user
        pass: !include secret.yaml#auth.pass

But it's only ok while the populated fields expect strings. But this is not always the case.

For example following will not work:

      host: !include secret.yaml#loc.host
      port: !include secret.yaml#loc.port

given that port is an int:

public sealed class Location
{
   public string host { get; set; }
   public int port { get; set; }
}

One workaround here would be to substitute the int port with a wrapper class having a Parse method:

public sealed class IntWrapper
{
   public int value { get; set; }

   public static IntWrapper Parse(string v) { 
      return new IntWrapper() {
         value = Int32.Parse(v)
      };
   }
}

public sealed class Location
{
   public string host { get; set; }
   public IntWrapper port { get; set; }
}

But this is kinda ugly and troublesome.

Also this means that there's no actual way to include structured classes (without manually re-inventing parsing for them second time). For example instead of this

      location: http://some.example/api
      auth:
        user: !include secret.yaml#auth.user
        pass: !include secret.yaml#auth.pass

i'd like to use

      location: http://some.example/api
      auth: !include secret.yaml#auth

So it would be really nice to have either a way to provide INodeDeserializer.Deserialize() with expected type info.

OR have a standard way to inject parts of external yaml streams into the main parser at specified places. Maybe some magic with MergingParser?

[EdwardCooke]

You would use a custom IYamlTypeConverter. I'll leave it up to you on reading the file. Doing it this way won't care about type of the scalar, number, string, whatever. Up to you however you want to parse your !include scalar.

using YamlDotNet.Core;
using YamlDotNet.Core.Events;
using YamlDotNet.Serialization;
using YamlDotNet.Serialization.NamingConventions;

var deserializer = new DeserializerBuilder()
    .WithTagMapping("!include", typeof(IncludedObject))
    .WithTypeConverter(new TestTypeConverter())
    .WithNamingConvention(CamelCaseNamingConvention.Instance)
    .Build();

var yaml = @"
location: http://some.example/api
auth: !include secret.yaml#auth
";

var o = deserializer.Deserialize<Outer>(yaml);
Console.WriteLine(o.Location);
Console.WriteLine(o.Auth);
Console.WriteLine(o.Auth?.Username);
Console.WriteLine(o.Auth?.Password);

class Outer
{
    public string Location { get; set; }
    public Auth Auth { get; set; }
}

class TestTypeConverter : IYamlTypeConverter
{
    public bool Accepts(Type type) => type == typeof(IncludedObject);

    public object? ReadYaml(IParser parser, Type type)
    {
        var isValid = parser.TryConsume<Scalar>(out var scalar);

        if (!isValid || scalar is null)
        {
            throw new Exception("Not a valid exception");
        }

        var split = scalar.Value.Split('#');
        var deserializer = new DeserializerBuilder().Build();
        if (split.Length != 2)
        {
            throw new Exception($"Invalid format, missing type and/or filename, { scalar.Value} ");
        }

        if (split[1] == "auth")
        {
            var yaml = @"
user: testusername
pass: testpassword
";
            var result = deserializer.Deserialize<Auth>(yaml);
            return result;
        }
        else
        {
            throw new Exception($"Unknown type: {split[1]}");
        }

        throw new Exception("Unexpected failure");
    }

    public void WriteYaml(IEmitter emitter, object? value, Type type)
    {
        throw new NotImplementedException();
    }
}

class IncludedObject
{

}

class Auth
{
    [YamlMember(Alias = "user")]
    public string Username { get; set; }

    [YamlMember(Alias = "pass")]
    public string Password { get; set; }
}

Results in

http://some.example/api
Auth
testusername
testpassword

[EdwardCooke]

Did the above answer your question?

[mikhail-barg]

@EdwardCooke Hi! sorry for a late answer, I've missed notifications (

Getting back to your solution, what bugs me, is that line:

var result = deserializer.Deserialize<Auth>(yaml);

How do I know that it's the Auth object expected at the place of !include? If I get your solution right, it requires knowing expected type upfront, and therefore I would not be able to make both

      location: http://some.example/api
      auth:
        user: !include secret.yaml#auth.user
        pass: !include secret.yaml#auth.pass

and

      location: http://some.example/api
      auth: !include secret.yaml#auth

to work?

[mikhail-barg]

It seems that a solution would be to provide an expected type via the type argument to ReadYaml call (or a new argument), so it would be possible to do

var result = deserializer.Deserialize(type, yaml);

see also this comment: #368 (comment)

[EdwardCooke]

If you don’t know the type you could just call deserialize with the yaml itself, no type. It’ll then return an object. Could be any number of types. An array, a string, int, short, etc. as well as a dictionary of object, object. Usually the key will be a primitive unless you have some crazy yaml where you make the key another mapping. This is not tested since I’m typing this on my phone but something like this may work.

var path = split[1].Split('.');
object last = deserializer.Deserialize(yaml);
foreach (var segment in path)
{
if (last is Dictionary<object, object> dict && dict.ContainsKey(segment))
{
last = dict[segment];
}
else
{
A throw new Exception(“key not found”);
}
}
return last;

[mikhail-barg]

@EdwardCooke thankjs for the answer! Still it seems that it does not solve the problem, if I get your solution right.

The problem is that the object model of the original document (I'm not sure I'm using correct terms here, so please bear with me) already expects a specific type, and if a type converter returns another tyle (something generic like string, object, or a dictionary like you proposed) there would just be a conversion error, correct?

For example, here's my DOM:

public sealed class Doc
{
   public string location { get; set; }
   public Auth auth { get; set; }
}

public sealed class Auth
{
   public string user { get; set; }
   public string pass { get; set; }
}

and for this yaml:

      location: http://some.example/api
      auth: !include secret.yaml#auth

returning a Dictionary for !include secret.yaml#auth would not work, because it's the Auth type that is expected here.

And currently there's no way to know that it should be an Auth type at the time IYamlTypeConverter.ReadYaml is executing (

[EdwardCooke]

Ok. So it would be a combination of the first 2. The if statement where it checks for split[1] would determine what to do. You may even be able to do this and not even worry about what’s passed in after the # sign.

deserializer.Deserialize(yaml, type);

yaml is the included file contents.

It’s the same as what you had in one of your comments just swap the arguments.

[EdwardCooke]

Just came up with a way to do what you are asking without needing to know the type before hand. It uses a custom nodedeserializer.

using YamlDotNet.Core;
using YamlDotNet.Core.Events;
using YamlDotNet.Serialization;
using YamlDotNet.Serialization.NamingConventions;
using YamlDotNet.Serialization.NodeTypeResolvers;

var deserializer = new DeserializerBuilder()
    .WithNodeDeserializer(new IncludeDeserializer(), syntax => syntax.OnTop())
    .WithoutNodeTypeResolver<PreventUnknownTagsNodeTypeResolver>()
    .WithNamingConvention(CamelCaseNamingConvention.Instance)
    .Build();

var yaml = @"
location: http://some.example/api
auth: !include secret.yaml#auth
";

var o = deserializer.Deserialize<Outer>(yaml);
Console.WriteLine(o.Location);
Console.WriteLine(o.Auth);
Console.WriteLine(o.Auth?.Username);
Console.WriteLine(o.Auth?.Password);

class Outer
{
    public string Location { get; set; }
    public Auth Auth { get; set; }
}

class IncludeDeserializer : INodeDeserializer
{
    public bool Deserialize(IParser reader, Type expectedType, Func<IParser, Type, object?> nestedObjectDeserializer, out object? value)
    {
        if (reader.Accept<Scalar>(out var scalar) && scalar.Tag == "!include")
        {
            var filename = scalar.Value.Split('#')[0];
            // Do your file check logic here
            if (filename == "secret.yaml")
            {
                reader.Consume<Scalar>();
                // read your yaml file
                var yaml = @"
user: testusername
pass: testpassword
";
                var deserializer = new DeserializerBuilder().Build();
                // deserialize to the object
                value = deserializer.Deserialize(yaml, expectedType);
                return true;
            }
        }
        value = null;
        return false;
    }
}

class Auth
{
    [YamlMember(Alias = "user")]
    public string Username { get; set; }

    [YamlMember(Alias = "pass")]
    public string Password { get; set; }
}

Results in

http://some.example/api
Auth
testusername
testpassword

[EdwardCooke]

If you want to make it pass correct validation (a comment I saw in aaubry's suggestion), use a custom type resolver instead of removing the PreventUnknownTagsNodeTypeResolver

var deserializer = new DeserializerBuilder()
    .WithNodeDeserializer(new IncludeDeserializer(), syntax => syntax.OnTop())
    .WithNodeTypeResolver(new IncludeNodeTypeResolver(), syntax => syntax.OnTop())
    .WithNamingConvention(CamelCaseNamingConvention.Instance)
    .Build();

class IncludeNodeTypeResolver : INodeTypeResolver
{
    public bool Resolve(NodeEvent? nodeEvent, ref Type currentType)
    {
        if (nodeEvent?.Tag == "!include")
        {
            return true;
        }
        return false;
    }
}

[mikhail-barg]

Thanks for the answer and samples! I am unable to check those right now, so I'll return with results a bit later..
Thanks again for your support and patience!

[EdwardCooke]

Did the above example work?

[EdwardCooke]

It's been a few weeks with a working example that answers the question. I'm going to close this issue. Open it again if you need further assistance.