You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
sdmx handles Observation as individual object instances. An alternative is to use pandas.DataFrame, xarray.DataSet, or other data structures internally.
See:
sdmx/experimental.py for a partial mock-up of such code, and
tests/test_experimental.py for tests.
Choosing either the current or experimental DataSet as a default should be based on detailed performance (memory and time) evaluation under a variety of use-cases.
The key question is: is there a performance limitation that this could solve?
To that end, note that the experimental DataSet involves three conversions:
a reader parses the XML or JSON source, creates Observation instances, and adds them using DataSet.add_obs()
experimental.DataSet.add_obs() populates a pd.DataFrame from these Observations, then discards them.
experimental.DataSet.obs() creates new Observation instances again.
For a fair comparison, the API between the readers and DataSet could be changed to eliminate the round trip in (1)/(2), but without sacrificing the data model consistency provided by pydantic on Observation instances.
The text was updated successfully, but these errors were encountered:
(Copied from the former doc/roadmap.rst.)
sdmx
handlesObservation
as individual object instances. An alternative is to use pandas.DataFrame, xarray.DataSet, or other data structures internally.See:
Choosing either the current or experimental DataSet as a default should be based on detailed performance (memory and time) evaluation under a variety of use-cases.
The key question is: is there a performance limitation that this could solve?
To that end, note that the experimental DataSet involves three conversions:
For a fair comparison, the API between the readers and DataSet could be changed to eliminate the round trip in (1)/(2), but without sacrificing the data model consistency provided by pydantic on Observation instances.
The text was updated successfully, but these errors were encountered: