Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DSP Improvements #324

Merged
merged 12 commits into from Jul 26, 2022
Merged

DSP Improvements #324

merged 12 commits into from Jul 26, 2022

Conversation

iguinn
Copy link
Collaborator

@iguinn iguinn commented Jul 24, 2022

  • Made debug output for processing_chain to not output values in arrays (this was TMI)
  • lgdo Scalar writing in overwrite mode didn't work
  • Changed behavior of the chan_config option, which lets you pass a dict pointing from channel name to dsp config files:
    • Before, you were required to provide a file for every channel in a file, and build_dsp would try to build every channel unless you use the lh5_tables option as well
    • Now, it loops through the dict and only builds provided channels; if it doesn't find a channel, it will skip it
    • Before you had to give the channel in chan_config as chan#/raw; now it will work with just chan#
  • Fixed progress bar bug that was causing it to misreport the number of events in a file when it completed
  • For non-integer pickoff times, instead of raising an exception, interpolate linearly between samples. This is important if we want to use pre-summing

@codecov-commenter
Copy link

codecov-commenter commented Jul 24, 2022

Codecov Report

Merging #324 (3e31ac9) into main (8673620) will decrease coverage by 0.00%.
The diff coverage is 6.25%.

@@            Coverage Diff             @@
##             main     #324      +/-   ##
==========================================
- Coverage   33.11%   33.11%   -0.01%     
==========================================
  Files          73       73              
  Lines        6184     6185       +1     
==========================================
  Hits         2048     2048              
- Misses       4136     4137       +1     
Impacted Files Coverage Δ
src/pygama/dsp/_processors/fixed_time_pickoff.py 0.00% <0.00%> (ø)
src/pygama/dsp/processing_chain.py 12.29% <0.00%> (ø)
src/pygama/dsp/build_dsp.py 16.49% <4.54%> (ø)
src/pygama/lgdo/lh5_store.py 76.13% <33.33%> (-0.18%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8673620...3e31ac9. Read the comment docs.

@iguinn iguinn requested a review from jasondet July 24, 2022 17:58
a_out[0] = w_in[int(t_in)]
i = int(t_in)
w = t_in-i
a_out[0] = (1-2)*w_in[i] + w*w_in[i+1]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is (1-2) doing there? Should it be (1-w)?

Copy link
Collaborator

@jasondet jasondet Jul 24, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also, I think it would be clearer to rename t_in as i_in since it is an index, not a time...

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

D'oh, fixed!

@@ -622,8 +622,9 @@ def write_object(self,

# scalars
elif isinstance(obj, Scalar):
if name in group:
log.debug(f"overwriting '{name}' in '{group}'")
if wo_mode == 'o' and name in group:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also feel like if name is in group and wo_mode is 'w' then we should emit an error here. @iguinn do you agree?

Also wo_mode = 'a' should be okay for overwriting scalars, according to the write_object` docstring.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made this change.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, sounds good. I guess the other case is 'of' but in that case I think no check is needed since the file's overwritten so it shouldn't be possible to run into this? Either way, raising an error ourselves is a good idea, as before it was raising a somewhat confusing h5py error.

@jasondet
Copy link
Collaborator

Finally @iguinn can you check why the Build documentation test is failing? I couldn't figure it out from the Details

@mmatteo
Copy link
Collaborator

mmatteo commented Jul 25, 2022

@iguinn, I don't like the idea of changing the behaviour of the processor and like to have it as it is now. I would rather implement a specific processor that supports different types of interpolation and can replace fixed_time_pickoff if the user wants so

@jasondet
Copy link
Collaborator

As of Ian’s PR, the code does not change behavior based on the cast of an input parameter, it does an interpolation every time. To my mind, this is the desired behavior of this processor. It gives consistent output independent of pre-summing, for example.

@iguinn
Copy link
Collaborator Author

iguinn commented Jul 25, 2022

Ok, I"ve reverted the changes to fixed_time_pickoff. I'll create an issue for it, but until we fix it we shouldn't start using presumming

@iguinn
Copy link
Collaborator Author

iguinn commented Jul 25, 2022

Finally @iguinn can you check why the Build documentation test is failing? I couldn't figure it out from the Details

I can't tell either, the error message is pretty unhelpful here...It looks to me like other pull requests are running into the same issue, though, so it's not clear that changes here are the cause. If this PR is actually the cause, I did make changes to a function signature (specifically an annotation) on line 26 of build_dsp.py, so it's possible that could have caused it?

@jasondet
Copy link
Collaborator

The build_documentation failure is a bug in sphinx. I'm going to turn off this requirement for merging and merge.

@jasondet jasondet merged commit 16a3246 into legend-exp:main Jul 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants