Uploading PROTOBUF_NATIVE fails way too hard :(

[vutran1710]

I tried to create a schema over REST API like this and the server just crash. Please help as I have been struggling with it for the last 24 hours:

My protobuf:

syntax = "proto3";

message User {
    string name = 1;
    int32 age = 2;
}

Below is the payload

{
  "type": "PROTOBUF_NATIVE",
  "schema": "syntax = \"proto3\";\n\nmessage User {\n    string name = 1;\n    int32 age = 2;\n}",
  "properties": {}
}

And here is error log:

com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'syntax': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
2024-04-10 22:14:09  at [Source: (byte[])"syntax = "proto3";
2024-04-10 22:14:09 
2024-04-10 22:14:09 message User {
2024-04-10 22:14:09     string name = 1;
2024-04-10 22:14:09     int32 age = 2;
2024-04-10 22:14:09 }"; line: 1, column: 8]
2024-04-10 22:14:09     at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2418)

[visortelle]

Hi @vutran1710. I agree, it's complicated for PROTOBUF_NATIVE

The schema should look like this:

{
  "version": 0,
  "schemaInfo": {
    "name": "commands",
    "schema": "{\"fileDescriptorSet\":\"CvYNCjV0b29scy90ZWFsL3B1bHNhci9kZW1vYXBwL3dhcmVob3VzZS92MS9jb21tYW5kcy5wcm90bxImdG9vbHMudGVhbC5wdWxzYXIuZGVtb2FwcC53YXJlaG91c2UudjEaKnRvb2xzL3RlYWwvcHVsc2FyL2RlbW9hcHAvZHRvL3YxL2R0by5wcm90byKuAQoUUmVjZWl2ZUludmVudG9yeUl0ZW0SIQoMaW52ZW50b3J5X2lkGAEgASgJUgtpbnZlbnRvcnlJZBJDCgdwcm9kdWN0GAIgASgLMikudG9vbHMudGVhbC5wdWxzYXIuZGVtb2FwcC5kdG8udjEuUHJvZHVjdFIHcHJvZHVjdBISCgRjb3N0GAMgASgJUgRjb3N0EhoKCHF1YW50aXR5GAQgASgFUghxdWFudGl0eSKJAQoXSW5jcmVhc2VJbnZlbnRvcnlBZGp1c3QSIQoMaW52ZW50b3J5X2lkGAEgASgJUgtpbnZlbnRvcnlJZBIXCgdpdGVtX2lkGAIgASgJUgZpdGVtSWQSGgoIcXVhbnRpdHkYAyABKAVSCHF1YW50aXR5EhYKBnJlYXNvbhgEIAEoCVIGcmVhc29uIokBChdEZWNyZWFzZUludmVudG9yeUFkanVzdBIhCgxpbnZlbnRvcnlfaWQYASABKAlSC2ludmVudG9yeUlkEhcKB2l0ZW1faWQYAiABKAlSBml0ZW1JZBIaCghxdWFudGl0eRgDIAEoBVIIcXVhbnRpdHkSFgoGcmVhc29uGAQgASgJUgZyZWFzb24i0gEKFFJlc2VydmVJbnZlbnRvcnlJdGVtEiEKDGludmVudG9yeV9pZBgBIAEoCVILaW52ZW50b3J5SWQSHQoKY2F0YWxvZ19pZBgCIAEoCVIJY2F0YWxvZ0lkEhcKB2NhcnRfaWQYAyABKAlSBmNhcnRJZBJDCgdwcm9kdWN0GAQgASgLMikudG9vbHMudGVhbC5wdWxzYXIuZGVtb2FwcC5kdG8udjEuUHJvZHVjdFIHcHJvZHVjdBIaCghxdWFudGl0eRgFIAEoBVIIcXVhbnRpdHkiTwoPQ3JlYXRlSW52ZW50b3J5EiEKDGludmVudG9yeV9pZBgBIAEoCVILaW52ZW50b3J5SWQSGQoIb3duZXJfaWQYAiABKAlSB293bmVySWQi9AQKF1dhcmVob3VzZUNvbW1hbmRzU2NoZW1hEnQKFnJlY2VpdmVfaW52ZW50b3J5X2l0ZW0YASABKAsyPC50b29scy50ZWFsLnB1bHNhci5kZW1vYXBwLndhcmVob3VzZS52MS5SZWNlaXZlSW52ZW50b3J5SXRlbUgAUhRyZWNlaXZlSW52ZW50b3J5SXRlbRJ9ChlpbmNyZWFzZV9pbnZlbnRvcnlfYWRqdXN0GAIgASgLMj8udG9vbHMudGVhbC5wdWxzYXIuZGVtb2FwcC53YXJlaG91c2UudjEuSW5jcmVhc2VJbnZlbnRvcnlBZGp1c3RIAFIXaW5jcmVhc2VJbnZlbnRvcnlBZGp1c3QSfQoZZGVjcmVhc2VfaW52ZW50b3J5X2FkanVzdBgDIAEoCzI/LnRvb2xzLnRlYWwucHVsc2FyLmRlbW9hcHAud2FyZWhvdXNlLnYxLkRlY3JlYXNlSW52ZW50b3J5QWRqdXN0SABSF2RlY3JlYXNlSW52ZW50b3J5QWRqdXN0EnQKFnJlc2VydmVfaW52ZW50b3J5X2l0ZW0YBCABKAsyPC50b29scy50ZWFsLnB1bHNhci5kZW1vYXBwLndhcmVob3VzZS52MS5SZXNlcnZlSW52ZW50b3J5SXRlbUgAUhRyZXNlcnZlSW52ZW50b3J5SXRlbRJkChBjcmVhdGVfaW52ZW50b3J5GAUgASgLMjcudG9vbHMudGVhbC5wdWxzYXIuZGVtb2FwcC53YXJlaG91c2UudjEuQ3JlYXRlSW52ZW50b3J5SABSD2NyZWF0ZUludmVudG9yeUIJCgdjb21tYW5kQvoBCipjb20udG9vbHMudGVhbC5wdWxzYXIuZGVtb2FwcC53YXJlaG91c2UudjFCDUNvbW1hbmRzUHJvdG9QAaICBVRUUERXqgImVG9vbHMuVGVhbC5QdWxzYXIuRGVtb2FwcC5XYXJlaG91c2UuVjHKAiZUb29sc1xUZWFsXFB1bHNhclxEZW1vYXBwXFdhcmVob3VzZVxWMeICMlRvb2xzXFRlYWxcUHVsc2FyXERlbW9hcHBcV2FyZWhvdXNlXFYxXEdQQk1ldGFkYXRh6gIrVG9vbHM6OlRlYWw6OlB1bHNhcjo6RGVtb2FwcDo6V2FyZWhvdXNlOjpWMWIGcHJvdG8zCv8BCh9nb29nbGUvcHJvdG9idWYvdGltZXN0YW1wLnByb3RvEg9nb29nbGUucHJvdG9idWYiOwoJVGltZXN0YW1wEhgKB3NlY29uZHMYASABKANSB3NlY29uZHMSFAoFbmFub3MYAiABKAVSBW5hbm9zQoUBChNjb20uZ29vZ2xlLnByb3RvYnVmQg5UaW1lc3RhbXBQcm90b1ABWjJnb29nbGUuZ29sYW5nLm9yZy9wcm90b2J1Zi90eXBlcy9rbm93bi90aW1lc3RhbXBwYvgBAaICA0dQQqoCHkdvb2dsZS5Qcm90b2J1Zi5XZWxsS25vd25UeXBlc2IGcHJvdG8zCo0eCip0b29scy90ZWFsL3B1bHNhci9kZW1vYXBwL2R0by92MS9kdG8ucHJvdG8SIHRvb2xzLnRlYWwucHVsc2FyLmRlbW9hcHAuZHRvLnYxGh9nb29nbGUvcHJvdG9idWYvdGltZXN0YW1wLnByb3RvIjsKBU1vbmV5EhYKBmFtb3VudBgBIAEoAVIGYW1vdW50EhoKCGN1cnJlbmN5GAIgASgJUghjdXJyZW5jeSLcAQoHQWRkcmVzcxIWCgZzdHJlZXQYASABKAlSBnN0cmVldBISCgRjaXR5GAIgASgJUgRjaXR5EhQKBXN0YXRlGAMgASgJUgVzdGF0ZRIZCgh6aXBfY29kZRgEIAEoCVIHemlwQ29kZRIYCgdjb3VudHJ5GAUgASgJUgdjb3VudHJ5EhsKBm51bWJlchgGIAEoA0gAUgZudW1iZXKIAQESIwoKY29tcGxlbWVudBgHIAEoCUgBUgpjb21wbGVtZW50iAEBQgkKB19udW1iZXJCDQoLX2NvbXBsZW1lbnQirwEKCkNyZWRpdENhcmQSQwoPZXhwaXJhdGlvbl9kYXRlGAEgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcFIOZXhwaXJhdGlvbkRhdGUSFgoGbnVtYmVyGAIgASgJUgZudW1iZXISHwoLaG9sZGVyX25hbWUYAyABKAlSCmhvbGRlck5hbWUSIwoNc2VjdXJpdHlfY29kZRgEIAEoBVIMc2VjdXJpdHlDb2RlIq4BCglEZWJpdENhcmQSQwoPZXhwaXJhdGlvbl9kYXRlGAEgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcFIOZXhwaXJhdGlvbkRhdGUSFgoGbnVtYmVyGAIgASgJUgZudW1iZXISHwoLaG9sZGVyX25hbWUYAyABKAlSCmhvbGRlck5hbWUSIwoNc2VjdXJpdHlfY29kZRgEIAEoBVIMc2VjdXJpdHlDb2RlIjoKBlBheVBhbBIUCgVlbWFpbBgBIAEoCVIFZW1haWwSGgoIcGFzc3dvcmQYAiABKAlSCHBhc3N3b3JkItYCCg1QYXltZW50TWV0aG9kEg4KAmlkGAEgASgJUgJpZBI/CgZhbW91bnQYAiABKAsyJy50b29scy50ZWFsLnB1bHNhci5kZW1vYXBwLmR0by52MS5Nb25leVIGYW1vdW50Ek8KC2NyZWRpdF9jYXJkGAMgASgLMiwudG9vbHMudGVhbC5wdWxzYXIuZGVtb2FwcC5kdG8udjEuQ3JlZGl0Q2FyZEgAUgpjcmVkaXRDYXJkEkwKCmRlYml0X2NhcmQYBCABKAsyKy50b29scy50ZWFsLnB1bHNhci5kZW1vYXBwLmR0by52MS5EZWJpdENhcmRIAFIJZGViaXRDYXJkEkMKB3BheV9wYWwYBSABKAsyKC50b29scy50ZWFsLnB1bHNhci5kZW1vYXBwLmR0by52MS5QYXlQYWxIAFIGcGF5UGFsQhAKDnBheW1lbnRfb3B0aW9uIpcBCgdQcm9kdWN0EiAKC2Rlc2NyaXB0aW9uGAEgASgJUgtkZXNjcmlwdGlvbhISCgRuYW1lGAIgASgJUgRuYW1lEhQKBWJyYW5kGAMgASgJUgVicmFuZBIaCghjYXRlZ29yeRgEIAEoCVIIY2F0ZWdvcnkSEgoEdW5pdBgFIAEoCVIEdW5pdBIQCgNza3UYBiABKAlSA3NrdSK3AQoNSW52ZW50b3J5SXRlbRIOCgJpZBgBIAEoCVICaWQSIQoMaW52ZW50b3J5X2lkGAIgASgJUgtpbnZlbnRvcnlJZBJDCgdwcm9kdWN0GAMgASgLMikudG9vbHMudGVhbC5wdWxzYXIuZGVtb2FwcC5kdG8udjEuUHJvZHVjdFIHcHJvZHVjdBISCgRjb3N0GAQgASgJUgRjb3N0EhoKCHF1YW50aXR5GAUgASgFUghxdWFudGl0eSLsAQoLQ2F0YWxvZ0l0ZW0SDgoCaWQYASABKAlSAmlkEh0KCmNhdGFsb2dfaWQYAiABKAlSCWNhdGFsb2dJZBIhCgxpbnZlbnRvcnlfaWQYAyABKAlSC2ludmVudG9yeUlkEkMKB3Byb2R1Y3QYBCABKAsyKS50b29scy50ZWFsLnB1bHNhci5kZW1vYXBwLmR0by52MS5Qcm9kdWN0Ugdwcm9kdWN0EhIKBGNvc3QYBSABKAlSBGNvc3QSFgoGbWFya3VwGAYgASgBUgZtYXJrdXASGgoIcXVhbnRpdHkYByABKAVSCHF1YW50aXR5IsMBCghDYXJ0SXRlbRIOCgJpZBgBIAEoCVICaWQSQwoHcHJvZHVjdBgCIAEoCzIpLnRvb2xzLnRlYWwucHVsc2FyLmRlbW9hcHAuZHRvLnYxLlByb2R1Y3RSB3Byb2R1Y3QSGgoIcXVhbnRpdHkYAyABKAVSCHF1YW50aXR5EkYKCnVuaXRfcHJpY2UYBCABKAsyJy50b29scy50ZWFsLnB1bHNhci5kZW1vYXBwLmR0by52MS5Nb25leVIJdW5pdFByaWNlIsQBCglPcmRlckl0ZW0SDgoCaWQYASABKAlSAmlkEkMKB3Byb2R1Y3QYAiABKAsyKS50b29scy50ZWFsLnB1bHNhci5kZW1vYXBwLmR0by52MS5Qcm9kdWN0Ugdwcm9kdWN0EhoKCHF1YW50aXR5GAMgASgFUghxdWFudGl0eRJGCgp1bml0X3ByaWNlGAQgASgLMicudG9vbHMudGVhbC5wdWxzYXIuZGVtb2FwcC5kdG8udjEuTW9uZXlSCXVuaXRQcmljZSLAAQoHUHJvZmlsZRIdCgpmaXJzdF9uYW1lGAEgASgJUglmaXJzdE5hbWUSGwoJbGFzdF9uYW1lGAIgASgJUghsYXN0TmFtZRIUCgVlbWFpbBgDIAEoCVIFZW1haWwSPQoJYmlydGhkYXRlGAQgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcEgAUgliaXJ0aGRhdGWIAQESFgoGZ2VuZGVyGAUgASgJUgZnZW5kZXJCDAoKX2JpcnRoZGF0ZSLEBQoMU2hvcHBpbmdDYXJ0Eg4KAmlkGAEgASgJUgJpZBIfCgtjdXN0b21lcl9pZBgCIAEoCVIKY3VzdG9tZXJJZBIWCgZzdGF0dXMYAyABKAlSBnN0YXR1cxJXCg9iaWxsaW5nX2FkZHJlc3MYBCABKAsyKS50b29scy50ZWFsLnB1bHNhci5kZW1vYXBwLmR0by52MS5BZGRyZXNzSABSDmJpbGxpbmdBZGRyZXNziAEBElkKEHNoaXBwaW5nX2FkZHJlc3MYBSABKAsyKS50b29scy50ZWFsLnB1bHNhci5kZW1vYXBwLmR0by52MS5BZGRyZXNzSAFSD3NoaXBwaW5nQWRkcmVzc4gBARI9CgV0b3RhbBgGIAEoCzInLnRvb2xzLnRlYWwucHVsc2FyLmRlbW9hcHAuZHRvLnYxLk1vbmV5UgV0b3RhbBJMCg10b3RhbF9wYXltZW50GAcgASgLMicudG9vbHMudGVhbC5wdWxzYXIuZGVtb2FwcC5kdG8udjEuTW9uZXlSDHRvdGFsUGF5bWVudBJGCgphbW91bnRfZHVlGAggASgLMicudG9vbHMudGVhbC5wdWxzYXIuZGVtb2FwcC5kdG8udjEuTW9uZXlSCWFtb3VudER1ZRJACgVpdGVtcxgJIAMoCzIqLnRvb2xzLnRlYWwucHVsc2FyLmRlbW9hcHAuZHRvLnYxLkNhcnRJdGVtUgVpdGVtcxJYCg9wYXltZW50X21ldGhvZHMYCiADKAsyLy50b29scy50ZWFsLnB1bHNhci5kZW1vYXBwLmR0by52MS5QYXltZW50TWV0aG9kUg5wYXltZW50TWV0aG9kcxIdCgppc19kZWxldGVkGAsgASgIUglpc0RlbGV0ZWRCEgoQX2JpbGxpbmdfYWRkcmVzc0ITChFfc2hpcHBpbmdfYWRkcmVzcyLQAgoSTm90aWZpY2F0aW9uTWV0aG9kEg4KAmlkGAEgASgJUgJpZBI/CgVlbWFpbBgCIAEoCzInLnRvb2xzLnRlYWwucHVsc2FyLmRlbW9hcHAuZHRvLnYxLkVtYWlsSABSBWVtYWlsEjkKA3NtcxgDIAEoCzIlLnRvb2xzLnRlYWwucHVsc2FyLmRlbW9hcHAuZHRvLnYxLlNtc0gAUgNzbXMSRgoIcHVzaF93ZWIYBCABKAsyKS50b29scy50ZWFsLnB1bHNhci5kZW1vYXBwLmR0by52MS5QdXNoV2ViSABSB3B1c2hXZWISTwoLcHVzaF9tb2JpbGUYBSABKAsyLC50b29scy50ZWFsLnB1bHNhci5kZW1vYXBwLmR0by52MS5QdXNoTW9iaWxlSABSCnB1c2hNb2JpbGVCFQoTbm90aWZpY2F0aW9uX29wdGlvbiJPCgVFbWFpbBIYCgdhZGRyZXNzGAEgASgJUgdhZGRyZXNzEhgKB3N1YmplY3QYAiABKAlSB3N1YmplY3QSEgoEYm9keRgDIAEoCVIEYm9keSIxCgNTbXMSFgoGbnVtYmVyGAEgASgJUgZudW1iZXISEgoEYm9keRgCIAEoCVIEYm9keSI2CgdQdXNoV2ViEhcKB3VzZXJfaWQYASABKAlSBnVzZXJJZBISCgRib2R5GAIgASgJUgRib2R5Ij0KClB1c2hNb2JpbGUSGwoJZGV2aWNlX2lkGAEgASgJUghkZXZpY2VJZBISCgRib2R5GAIgASgJUgRib2R5QtcBCiRjb20udG9vbHMudGVhbC5wdWxzYXIuZGVtb2FwcC5kdG8udjFCCER0b1Byb3RvUAGiAgVUVFBERKoCIFRvb2xzLlRlYWwuUHVsc2FyLkRlbW9hcHAuRHRvLlYxygIgVG9vbHNcVGVhbFxQdWxzYXJcRGVtb2FwcFxEdG9cVjHiAixUb29sc1xUZWFsXFB1bHNhclxEZW1vYXBwXER0b1xWMVxHUEJNZXRhZGF0YeoCJVRvb2xzOjpUZWFsOjpQdWxzYXI6OkRlbW9hcHA6OkR0bzo6VjFiBnByb3RvMw\u003d\u003d\",\"rootMessageTypeName\":\"tools.teal.pulsar.demoapp.warehouse.v1.WarehouseCommandsSchema\",\"rootFileDescriptorName\":\"tools/teal/pulsar/demoapp/warehouse/v1/commands.proto\"}",
    "type": "PROTOBUF_NATIVE",
    "timestamp": 1713353523609,
    "properties": {
      "__PARSING_INFO__": "[{\"number\":1,\"name\":\"receive_inventory_item\",\"type\":\"MESSAGE\",\"label\":\"LABEL_OPTIONAL\",\"definition\":null},{\"number\":2,\"name\":\"increase_inventory_adjust\",\"type\":\"MESSAGE\",\"label\":\"LABEL_OPTIONAL\",\"definition\":null},{\"number\":3,\"name\":\"decrease_inventory_adjust\",\"type\":\"MESSAGE\",\"label\":\"LABEL_OPTIONAL\",\"definition\":null},{\"number\":4,\"name\":\"reserve_inventory_item\",\"type\":\"MESSAGE\",\"label\":\"LABEL_OPTIONAL\",\"definition\":null},{\"number\":5,\"name\":\"create_inventory\",\"type\":\"MESSAGE\",\"label\":\"LABEL_OPTIONAL\",\"definition\":null}]",
      "__alwaysAllowNull": "true",
      "__jsr310ConversionEnabled": "false"
    }
  }
}

I never used REST API or pulsar-admin for this. I use protoc to compile .proto files, protobuf-java to construct a descriptor that I can feed to ProtobufNativeSchemaUtils.serialize() method to get the resulting Pulsar schema .json I use for upload the schema 🤯
I also spent about a day to figure it all out the first time.

pulsar/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/ProtobufNativeSchemaUtils.java

Line 42 in d0b9d47

public static byte[] serialize(Descriptors.Descriptor descriptor) {

It works for me, but probably a simpler solution also exists.

[vutran1710]

Actually i have found the solution with the admin api by digging thru the test files in the codebase.

But its kinda a little different from your suggestion 😅

[vutran1710]

I still hope the team will update the document soon, bcs not all projects that use pulsar are developed with java/python.

[visortelle]

I still hope the team will update the document soon

Could you describe your solution in short or point me to it? It could help you, me, or someone else who would like to contribute by updating the docs.

Does it handle imports in proto files? It was one of the reasons the protoc was involved in my solution.

[vutran1710]

Sure will, as soon as i get to my computer.