Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Container types are inconsistent even within the same patch of Python protobuf #16411

Open
lazarillo opened this issue Apr 4, 2024 · 0 comments
Assignees

Comments

@lazarillo
Copy link

What version of protobuf and what language are you using?

Protobuf version: 4.25.3
Python version: 3.11

What operating system (Linux, Windows, ...) and version?

[local] Linux and [remote] Google Composer (Airflow) environment with image: composer-2.6.6-airflow-2.7.3

More details on this below.

What runtime / compiler are you using (e.g., python version or gcc version)

Python 3.11

What did you do?

When I create a protobuf message that contains a map type, for instance:

syntax = "proto3";

message OrderContract {
  map<string, string> custom = 1;
}

I am seeing that when working with it in Python, on my local machine (which is Ubuntu linux, also using Python 3.11, more details below) the custom element is registered as a google._upb._message.ScalarMapContainer, but on the remote Composer (Airflow) environment, the custom element is registered as a google.protobuf.internal.containers.ScalarMap.

Both of these environments are as exactly matched as possible: both are unix, both are using protobuf version 4.25.3, and both are using Python 3.11.

Problem

I need to perform introspection: I am creating some fake data, generating it using the faker library when I have hit a primitive element (str, int, etc), but introspecting further to get to a lower leaf when I have a protobuf Message, ScalarMapContainer, or some other protobuf (non-primitive) type.

It is difficult to debug this introspection since there is inconsistency even across the same Major.Minor.Patch.

I understand that these are supposed to be internals, but with no way to perform introspection, I am forced to work this way. I simply need to understand when and why google._upb._message... is used over google.protobuf.internal.containers.... I read that this happened with the change to protobuf 4.*, but this is clearly not the case, since I have the exact same version in both cases, and they are using the two different underlying libraries.

Anything else we should know about your project / environment

My linux machine details:

> uname -a
Linux <my-account> 6.5.0-26-generic #26~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Mar 12 10:22:43 UTC 2 x86_64 x86_64 x86_64 GNU/Linux`

As mentioned above, the remote machine is a Google Cloud Composer image, version composer-2.6.6-airflow-2.7.3. It's details can be found here.


Here is an output of python -m pip show protobuf across the two environments:

My local:

Name: protobuf
Version: 4.25.3
Summary: 
Home-page: https://developers.google.com/protocol-buffers/
Author: protobuf@googlegroups.com
Author-email: protobuf@googlegroups.com
License: 3-Clause BSD License
Location: /home/<my-account>/.local/lib/python3.11/site-packages
Requires: 
.
.
.

Cloud Composer:

Name: protobuf
Version: 4.25.3
Summary: 
Home-page: https://developers.google.com/protocol-buffers/
Author: protobuf@googlegroups.com
Author-email: protobuf@googlegroups.com
License: 3-Clause BSD License
Location: /opt/python3.11/lib/python3.11/site-packages
Requires: 
.
.
.

@lazarillo lazarillo added the untriaged auto added to all issues by default when created. label Apr 4, 2024
@shaod2 shaod2 added python upb and removed untriaged auto added to all issues by default when created. labels Apr 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants