-
Notifications
You must be signed in to change notification settings - Fork 11
/
CHANGELOG
1353 lines (1353 loc) · 59.5 KB
/
CHANGELOG
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
0.59.1
- fix: protection against cyclic basin dependencies
- fix: ValueError when exporting data from basin-mapped dataset
- fix: Check for basin availability when checking run identifier
0.59.0
- feat: support basins with blown indices
- enh: increase verbosity when failing to resolve basins
- enh: implement dtype for `H5ScalarEvent`
0.58.8
- enh: existence of "index_online" section is no longer required
0.58.7
- enh: introduce `RTDCBase.features_local` for accessing features
that are exclusively local and not in remote locations
- enh: prevent `RTDCBase._get_ancillary_feature_data` from
unnecessarily accessing feature data for hashing
- enh: introduce new feature "bg_off", a float-valued feature for
event-wise background offset correction
- enh: warn user about missing endpoint URLs in S3 format and raise an
exception when push comes to shove
- enh: add "pipeline:dcnum mapping" metadata
- docs: document S3 environment variables
0.58.6
- enh: for access to private S3 data, introduce the environment
variables `DCLAB_S3_ENDPOINT_URL`, `DCLAB_S3_ACCESS_KEY_ID`,
and `DCLAB_S3_SECRET_ACCESS_KEY`
- ref: reorganize the `fmt_s3` module
0.58.5
- fix: CLI methods returned exit status 1 since paths were returned
0.58.4
- fix: support relative paths in basins with both nt `\\` and
posix `/` path separators
- enh: store original metadata in .tsv export (#255)
0.58.3
- fix: keep only 1000 contours in LazyContourList, because keeping
all contours causes the memory to fill up for large input files
0.58.2
- fix: KeyError when iteration over DCOR logs or tables
- fix: remove hack that populates feature names for basins unreliably
- enh: lazily load logs and tables in `fmt_hierarchy`
0.58.1
- fix: exporting with basins from a hierarchy child did not write a
basin derived from the hierarchy root parent
- docs: add example for `map_indices_child2parent` in `fmt_hierarchy`
- ref: explicitly define ancillary features in `RTDC_Hierarchy`
0.58.0
- feat: implement "mapped basins" that have a superset of events
- feat: allow to export basin-only HDF5 files
- fix: support input files without "events" in `rtdc_copy`
- fix: CLI methods now store basin information in output paths (#239):
compress, condense, and repack preserve the original input basins;
split writes mapped basins (using export.hdf5); join does not write
basins at all
- fix: enable nested basins for the HDF5, S3, and HTTP basins,
because we have the `_local_basins_allowed` property since 0.57.6
- fix: when joining datasets, the name of the log holding the configuration
of the source datasets should be src-#{i}_cfg instead of cfg_src-#{i}
- enh: introduce `RTDCBase.features_ancillary`, a list of all ancillary
features (computed on-the-fly) for a dataset
- enh: add `include_basins` keyword argument for `rtdc_copy`
- enh: `RTDCWriter.store_basin` now returns the basin hash/identifier
- enh: CLI methods now return the output path
- enh: the repack CLI method now allows to strip basins
- docs: add an advanced usage section on basins
- docs: update doc strings for CLI methods
- setup: pin nptdms<1.9, because tests started to fail
- ref: modify basin `load_dataset` methods to support mapped basins
- ref: put input and output path in order for CLI methods
- tests: make sure basin features are not written automatically in
CLI methods (#246)
- ci: install lme4 from archive
0.57.7
- fix: raise a ValueError in HTTPFile when the server does not return
the status code 200
- enh: properly handle missing ETag in HTTPFile
- enh: compute fallback identifier using hash of first chunk in RTDC_HTTP
- ref: drop the old dcserve API version 1 and use version 2 by default
0.57.6
- enh: allow non-existent paths in `common.get_command_log`
- ref: factor out `cli.task_condense.condense_dataset` for condensing
an open dataset without the knowledge of its path
- ref: when condensing a dataset, instead of the MD5-5M, the hash of its
configuration is used as an suffix for previous condense logs
0.57.5
- fix: do not allow local basins for network-based subclasses of `RTDCBase`
0.57.4
- fix: catch zero-valued contour accuracy for KDE plot (#249)
0.57.3
- setup: the s3 extra does not require a specific version of urllib3
- ci: use main branch of GH Actions actions to reduce maintenance burden
0.57.2
- fix: speed-up S3 availability check (#245)
- enh: allow to specify list of features for data copier
- ref: migrate from s3fs to boto3 for fmt_s3
- test: data copier with `none` and invalid features
- docs: remove pin for sphinx
- build: make musllinux builds for e.g. docker images
0.57.1
- fix: RTDCWriter.rectify_metadata fails when image feature is empty
- fix: handle empty write requests in export.hdf5 and RTDCWriter (#242)
- tests: add test for writing all-nan data (#242)
- docs: improve documentation of hierarchy child mapper
0.57.0
- fix: integer overflow in downsample_grid
- fix: removed unnecessary computation of hierarchy filter instance
- enh: cythonize downsample_grid (~10x speed-up)
- enh: got rid of for-loop in map_indices_parent2child (~100x speed-up)
- enh: slight improvement of managing manual indices in hierarchy children
- enh: added dtype properties for contour and trace events
- enh: ensure all feature data objects have the dtype property
- enh: globally use chunk sizes of ~1MiB when writing HDF5 data (#217)
(minor speedup since previously a chunk size of 100 events was used
for images and scalar features were written in one big chunk)
- enh: favor array operations in `yield_filtered_array_stacks` (~4x speed-up)
- enh: increase verbosity for unreachable remote basin features
- ref: implement `__array__` methods for hierarchy event classes
- ref: implement `__array__` method for `H5MaskEvent`
- ref: new submodule for hierarchy format
0.56.3
- fix: regression missing check for basin availability
0.56.2
- enh: perform basin availability checks in daemon thread
0.56.1
- enh: priority-based basin sorting (file over remote, http over dcor)
0.56.0
- feat: allow nested basins
- feat: implement DCOR basins
- fix: make sure basins are always closed on context exit (#238)
- fix: failed to load basin data when port was specified in location
- enh: requests session pooling for fmt_http and fmt_dcor
- enh: implement context manager for RTDCBase
- enh: be forwards-compatible and ignore unsupported basin formats
- ref: new http_utils submodule for managing HTTP connections
0.55.7
- test: tested rtdc_dataset.config.Configuration is pickable (#190)
- enh: add dcnum pipeline identifier metadata constants
0.55.6
- fix: default `host` was overriding URL host in fmt_dcor
0.55.5
- enh: register new features area_um_raw, deform_raw, eccentr_prnc,
per_ratio, per_um_raw, sym_x, sym_y
- ref: simplify/cleanup computation of tilt feature
0.55.4
- enh: improve timeout and availability check for fmt_http
0.55.3
- enh: migrate to purely requests-based HTTP file for fmt_http
0.55.2
- fix: fmt_dcor did not properly retry failed connections
- enh: minor optimizations for fmt_http and fmt_s3
0.55.1
- fix: disable instance cache fsspec.HTTPFileSystem (fmt_http)
0.55.0
- feat: implement HDF5-based HTTP file format and basin
- fix: replace lru_cache on instance methods to fix memory leak (#214)
- fix: remove lru_cache in DCOR format (#226)
- fix: implement `__contains__` for DCOR logs and tables
- enh: improve parsing of DCOR URLs (scheme in host)
- enh: speed-up of initialization of DCOR data
- enh: add requests timeout for DCOR data
- enh: more caching of event size and shape for HDF5 format
- enh: faster computation of contour length for DCOR format
- enh: use dcserv version 2 in DCOR format (fast S3 access)
- enh: change measurement identifier upon filtered export
- setup: pin s3fs>=2023.10.0
- setup: pin upper bounds of dependencies
0.54.2
- fix: availability checks for basins
0.54.1
- fix: properly check for basin availability before getting features
0.54.0
- fix: partial workaround for #238
- fix: hash alias for HDF5 Datasets in files not in the file system
- enh: delayed computation of measurement identifiers for basins
- enh: allow to specify which features are served by a basin
- enh: improve speed of S3 file format by increasing chunk size
- enh: do not immediately raise an exception when metadata are missing
- enh: lazy initialization of .RTDCBase.filters
- enh: lazy initialization of .RTDCBase.basins
- enh: lazy loading of feature list for HDF5 data
- setup: migrate from pkg_resources to importlib_resources
- ref: remove the dclab.isoelastics.ISOFILES constant
- ref: remove the dclab.features.emodulus.INTERNAL_LUTS constant
0.53.3
- fix: bad hdf5 basin check
0.53.2
- fix: catch OSError when accessing invalid hdf5 basin
- enh: implement retrieving basins in fmt_dcor
0.53.1
- fix: catch OSError when accessing invalid S3 URL
- enh: implement `Configuration.as_dict`
- enh: implement `Basin.as_dict`
0.53.0
- feat: introduce remote type basin format s3
- feat: implement `store_basin` in `RTDCWriter`
- docs: add information about basins
- ref: expand `fmt_dcor` submodule laterally
- ref: deprecate the `feat_anc_ml` submodule
- enh: improve URL parsing for s3 format
- enh: extract dataset size from metadata before resorting to features
- enh: support remote basins
0.52.6
- maintenance release
0.52.5
- maintenance release
0.52.4
- ref: HDF5Basin gets its own submodule
- setup: oldest-supported-numpy is not required anymore, since numpy
is backwards-compatible now
0.52.3
- maintenance release
0.52.2
- maintenance release
0.52.1
- docs: document the copier submodule (#218)
- docs: simplify headings in advanced section
- docs: refactor writing DC data into its own subsection
- docs: add advanced section on how to work with S3 data (#228)
- docs: add information about S3 object storage in DCOR section
- setup: force requests >= 2.31.0 (CVE-2023-32681)
- setup: bump scipy to 1.10.0 (CVE-2023-25399)
- tests: add test tables and logs tests for linker
- tests: add tests for S3 file format (#228)
0.52.0
- feat: qpi features and related metadata added to definitions (#192)
0.51.4
- setup: wrong package setup
0.51.3
- setup: wrong package setup
0.51.2
- enh: support dictionary of arrays in RTDCWriter.store_table
- tests: added missing test files
0.51.1
- fix: workaround for segmentation fault caused by HDF5
0.51.0
- feat: introduce read-support for local basins (upstream features)
- fix: minor bug lead to issue with latest rpy2 versions (invisible)
- fix: support numpy integers and booleans for JSON serializer
- enh: introduce optional configuration key [experiment]:"run identifier"
- docs: restructuring and introduction of new features
- ref: restructured fmt_hdf5 submodule
- ref: remove deprecated `write` method for creating .rtdc files
- ref: introduce more general `RTDCBase._finalize_init`
- tests: tests passing on Python 3.11
0.50.4
- setup: cleanup pyproject.toml
0.50.3
- fix: do not allow editing reserved filter properties (#210)
- fix: do not allow editing temporary features (#202)
- ref: read-only features and proper subclassing in `RTDC_Dict`
- ref: subclass from `collections.UserDict` and not from `dict`
- docs: add more information about manual filters (#173)
0.50.2
- enh: support reading 'tables' from DCOR datasets (#221)
0.50.1
- enh: support reading 'logs' from DCOR datasets
- enh: hierarchy parents inherit their parent's logs and tables
0.50.0
- feat: support opening objects on S3-compatible storage (#213)
- feat: support opening file-like objects as HDF5 data
- enh: use regexp to identify valid DCOR URLs
- enh: allow to pass keyword arguments to h5py.File for fmt_hdf5
0.49.1
- fix: do not recompute inert_ratio_cvx and inert_ratio_raw
computed with Shape-In >= 2.0.5 (#224)
- tests: add additional test for inertia ratio (#223)
0.49.0
- feat: implement `is_properly_compressed`, `rtdc_copy`, and
`h5ds_copy` in `dclab.rtdc_dataset.copier` (#216)
- feat: support "tables" (compound arrays) in HDF5 files
- fix: explicitly use 64bit ints and floats when instantiating numpy
arrays, because numpy on Windows defaults to 32bit ints
which resulted in overflows when computing inertia ratio
- enh: unify names of exported logs
- enh: CLI use `rtdc_copy` in dclab-compress, dclab-condense,
and dclab-repack (#216)
- enh: CLI compute the command log as early as possible
- enh: ignore "def" feature when performing an unknown-feature check
- enh: minor improvements in error message verbosity here and there
- ref: remove deprecated method dclab.load.check_dataset
0.48.8
- enh: CLI new --no-ancillary-features keyword argument for
dclab-condense to skip the computation of expensive features
0.48.7
- maintenance release
0.48.6
- maintenance release
0.48.5
- fix: CLI dclab-verify-datasets now complains about missing input data
- enh: feed kwargs to sp.check_output to avoid pop-up command prompts
- ref: migrate from "tensorflow-SavedModel" to "tf" in ml submodule
- build: migrate to pyproject.toml (except for cython extensions)
- build: add manylinux wheels using cibuildwheel
0.48.4
- fix: do not cache all feature names, only defective feature names,
because some features can be (de)registered dynamically
0.48.3
- fix: always treat ancillary time as float64
- enh: add defective feature check for inertia ratio features
inert_ratio_cvx, inert_ratio_prnc, inert_ratio_raw, and tilt (#212)
- enh: cache available feature names in the HDF5 format
0.48.2
- fix: invalid computation of inertia ratio features (inert_ratio_cvx,
inert_ratio_prnc, inert_ratio_raw, tilt) for events with large
horizontal extent due to an integer overflow (#212)
- enh: many-fold increase in HDF5 dataset read speed via caching (#189)
- enh: detect all-zero temperature in dclab-verify-dataset (#183)
0.48.1
- fix: emodulus values were not correctly cached
0.48.0
- BREAKING CHANGE: remove deprecated "emodulus model" key from
dataset configuration
- feat: new 'buyukurganci-2022' model to compute MC-PBS viscosity (#197)
- enh: use the 'buyukurganci-2022' model for isoelastics by default
- enh: increase verbosity when opening an .rtdc file failed
- enh: support shorthand for known media for viscosity computation
- docs: update part about DCOR (#182)
0.47.8
- fix: implement cleaner solution for time computation (#207)
0.47.7
- fix: include time offset when computing defective time feature (#207)
- ref: make the h5py.File object public in RTDC_HDF5
0.47.6
- fix: register float32 time from ShapeIn as defective feature
- docs: make "edit on GitHub" link lead to actual source file again
0.47.5
- enh: allow temporary features for hierarchy children (#123)
- enh: more type annotations (partially #198)
- build: add Python 3.11 CI pipeline
0.47.4
- enh: allow pathlib.Path objects in cli.get_command_log custom dicts
0.47.3
- fix: np.histrogram2d does not expect broken normed argument
- docs: fixed GH Actions badge
- enh: implement RTDC_HDF5.__len__
0.47.2
- enh: add --version flag to CLI
0.47.1
- fix: matplotlib.plot hierarchy child scalar data not working
0.47.0
- feat: lazy-loading and improved caching strategy for hierarchy
child datasets via `RTDC_Hierarchy` (the call to `apply_filter`
is now mandatory for all changes to propagate to the children)
- feat: introduce `RTDC_Hierarchy.rejuvenate`, an alias for
`apply_filter` which more accurately describes what happens there
- feat: added file_monitoring_lru_cache convenience decorator
- enh: new feature bg_med: Median frame background brightness
- enh: add min/max/mean values as scalar dataset attributes (#175)
0.46.6
- fix: tdms format config parser not working when para.ini missing
0.46.5
- enh: new staticmethod RTDC_TDMS.extract_tdms_config for
extracting metadata from tdms-based datasets; This method
optionally returns a list of paths used for extracting metadata
- ref: cleanup in RTDC_TDMS with dict.setdefault
0.46.4
- enh: allow to export logs alongside HDF5 data (#180)
- enh: add features flow_rate and pressure
0.46.3
- docs: make compilation run on Windows
- docs: update information on plugin features (#151)
- ref: remove Zellmechanik Dresden branding
0.46.2
- fix: support passing instances of `Configuration` and
`ConfigurationDict` to `dclab.util.obj2bytes`
- enh: return sorted json representation in `Configuration.tojson`
- ref: use `functools.update_wrapper` in `Cache` wrapper
- docs: make sure `dclab.downsampling.downsample_grid` is in code ref
0.46.1
- fix: ConfigurationDict should not subclass from dict (#174)
- ref: use f-strings and print correct feature for temporary feature
error message (#193)
0.46.0
- BREAKING CHANGE: The LUT data for LE-2D-FEM-19 had to be revised,
because the simulation support was sparse at high deformations (#191)
- feat: added new LUT and isoelastics for HE-2D-FEM-22
and HE-3D-FEM-22 (#188)
- docs: new LUT data version 10.6084/m9.figshare.12155064.v4
0.45.0
- feat: introduce 10th and 90th percentile brightness features
- enh: added definitions for the Haralick texture features
(computed using the mahotas package from bg-corrected event mask)
0.44.0
- feat: introduce background-corrected brightness features
- enh: detect confusion in plugin feature names (#179)
- enh: cache innate feature data for hierarchy children
- enh: raise more verbose error message when the user tries to access
non-existent features in hierarchy children (#92)
0.43.1
- fix: script fem2rtdc.py did not work with latest openCV (#176)
- fix: properly implement random pixel offsets in fem2rtdc.py and
and pixelation_correction.py (#178)
- enh: improved test for compression (require Zstd with level >= 5)
- enh: added `close` method for `RTDCWriter`
- ref: added tests and cleaned up fem2rtdc.py
0.43.0
- feat: introduce Zstandard compression via hdf5plugin
- enh: speed-up HDF5 data export if `filltarr` are all-True
- ref: deprecate "compression" keyword argument for HDF5 export
0.42.3
- enh: support hierarhcy feature dtype property for image and mask event
- docs: fix regression in builds due to new H5ScalarEvent
0.42.2
- fix: lazy-load scalar feature data for HDF5 data
- fix: fix repr string of AncillaryFeature when identifier is None
- ref: compute identifiers from HDF5 data using file system path
and HDF5 path
0.42.1
- fix: support "image_bg" feature for DCOR data
- ref: replace formats with f-strings in check.py
0.42.0
- enh: reduce computation time of file hashes in the cli submodule
(and thus all other software downstream) by switching from a full
SHA256 hash to a 5MB md5 hash; this speeds up operations on
slow network shares
- ref: minor change when installing R packages
(rlme tests are broken on Windows currently)
- ref: use np.float32 for principal inertia ratio computation
(compatibility with OpenCV)
0.41.0
- feat: pull DCOR access token management from downstream DCOR-Aid
- feat: allow to set alternate DCOR server certificate bundles by
appending paths `dclab.rtdc_dataset.fmt_dcor.DCOR_CERTS_SEARCH_PATHS`
- enh: allow to skip checks and by default use the innate features
during export of an RTDCBase
- setup: remove appveyor build pipeline
- ref: cleanup fmt_dcor
0.40.0
- setup: bump numpy from 1.17.0 to 1.21.0
- setup: bump scipy from 0.14.0 to 1.8.0
- setup: drop support for Python 3.7
0.39.18
- fix: regression since 0.39.0; contours with wrong index where not
repaired
- ref: Python distutils library is deprecated
- ref: some scipy namespaces are deprecated
0.39.17
- enh: allow to specify unique_id of PolygonFilter when loading
from file
0.39.16
- fix: not all statistics exported if no feature was specified
0.39.15
- fix: fix accessing inherited non-scalar temporary or plugin features
for RTDC_Hierarchy class (#165)
- fix: properly handle user-defined (temporary or plugin) features
when exporting to HDF5 (#166)
- enh: add "shape" keyword argument in RTDCWrite.store_feature to
allow the user to pass the correct data shape; this reduces warning
messages when using the RTDCBase.export.hdf5 functionality
- ref: simplify data export to HDF5 with generator function
- tests: use correct tdms-dataset for tdms tests
0.39.14
- fix: enable loading of non-scalar plugin data as h5py.Dataset (#162)
- build: minor fixes in CI and testing pipeline
0.39.13
- fix: recompute the "ml_class" feature if the ml_score features
change (either the underlying model or the number of features)
0.39.12
- fix: support nan values for the computation of the "ml_class"
feature (#161)
- fix: cache of util.hashfile used access times of files (and thus
might not have been of any use in some cases)
0.39.11
- fix: misleading error message when loading plugin features
with Python modules that are not available (#160)
- docs: add information on how to call CLI functions from within
Python (#145)
0.39.10
- fix: make absolutely sure HDF5 files are closed if RTDCWriter is
used as a context manager
- docs: properly document the shapes option of plugin features (#146)
0.39.9
- fix: spelling of Young's modulus (#159)
- ref: define minimal R version 3.6.0 in lme4 submodule and
propagate any errors with possible solutions to the user
- ref: extract exact version string for R
0.39.8
- enh: cache computed file hashes with additional os.stat info
- tests: fix tests on Windows
0.39.7
- fix: handle empty trace feature in RTDCWriter.rectify_metadata
0.39.6
- fix: do not close H5File object passed to RTDCWriter.__init__
0.39.5
- fix: write FeatureSetNotIdenticalJoinWarnings to logs in dclab-join
0.39.4
- fix: KeyError object 'index_online' doesn't exist when joining
two datasets with dclab-join (#158)
- fix: generalize online_filter configuration keys (#156)
- fix: relax validation of online_filter configuration key types
- fix: make sure that only common features of the input files for
dclab-join are written to the output file (#157)
- enh: add `with_unit` keyword to `get_feature_label`
- ref: slightly modified online_filter configuration descriptions
0.39.3
- ref: deprecate these lists/dicts in dclab.definitions:
config_descr, config_funcs, config_types, feature_names,
feature_labels, feature_name2label (#135)
- ref: use more intuitive trace feature in fmt_dcor
0.39.2
- ref: renamed mask and contour labels
- ref: increase verbosity in error messages
- tests: add .modc test model and data
0.39.1
- fix: don't create __pycache__ folders for plugin features
- fix: add "date" to machine-learning feature info dict
0.39.0
- enh: implement .shape and .__len__ for non-scalar features
and all file formats (#117)
- ref: adjust ancillary machine-learning feature API to match
that of plugin features (this is not a breaking change,
because nobody is using machine-learning features yet)
- ref: minor refactoring of tdms file format code might lead to
faster loading of trace data
0.38.2
- fix: reduce memory usage when writing "mask" feature data
0.38.1
- maintenance release
0.38.0
- enh: add identifier to AncillaryFeature class
- enh: introduce MachineLearningFeature to simplify the use of
machine learning models in ancillary features
- fix: load_modc should return a list of models
- ref: rename `rtdc_dataset.ancillary` submodule to `anc_feat_core`
- ref: rename `rtdc_dataset.plugins` submodule to `anc_feat_plugin`
- ref: move the `ml` submodule to `rtdc_dataset.anc_feat_ml`
- ref: move tensorflow-related code into its own submodule
- setup: bump minimum Python version to 3.7
0.37.3
- fix: ignore defective feature "volume" in pre 0.37.0 .rtdc files
- fix: always perform version branding when using RTDCWriter context
manager
- ref: silence numpy deprecation warning in RTDCWriter
0.37.2
- enh: speed-up contour computation by a factor of three
(volume is computed by factor of two faster)
0.37.1
- enh: make IntegrityChecker class available on top level module
0.37.0
- BREAKING CHANGE: The volume feature in all previous versions was
overestimated by about 2µm³. Please re-run your analysis pipeline.
- fix: volume was computed incorrectly (#141)
0.36.1
- setup: bump h5py version to 3.0 (issue with integer chunk argument)
0.36.0
- feat: introduce new RTDCWriter class (#144)
- fix: in some rare cases, an empty .rtdc file could be generated
when splitting an .rtdc file via the CLI dclab-split
- fix: user-defined metadata were not copied with CLI repack
- fix: support exporting non-scalar plugin features to HDF5 (#143)
- enh: implement consistent `.shape` property for contour data
(partially #117)
- enh: improve export speed of contour data
- enh: improve export speed if data filtering is disabled
- ref: disable `time_offset` of internal `export.hdf5_append` function
- ref: deprecate write_hdf5.write function
- ref: deprecate export.hdf5_autocomplete_config function
- ref: deprecate export.hdf5_append function
0.35.8
- fix: write logs to fixed-length strings in HDF5 files (HDF5 never
supported variable length string datasets with fletcher32 or compression
filters)
- fix: non-scalar features that are not defined in dclab were silently
ignored upon export; temporarily raise a ValueError instead (#143)
- setup: release pinning of h5py
0.35.7
- fix: more Shape-In versions are affected by the medium-write bug
addressed in 0.33.1
- enh: improve error handling when loading rpy2
- enh: add checks for negative or zero-valued metadata
- tests: fix broken example dataset with zero-valued metadata (#137)
0.35.6
- fix: config converters were not applied to raw metadata (#139)
0.35.5
- enh: add sanity_check method to IntegrityChecker
0.35.4
- enh: implement function for obtaining the description of a
configuration key get_config_value_descr
0.35.3
- enh: implement dumping of Configuration to json
0.35.2
- enh: allow to compress, condense, and repack without checking for
the input file suffix
- docs: fixed example data (sphinx encountered unexpected warnings)
0.35.1
- fix: ignore empty logs (raised IndexError before)
- tests: increase coverage (#136)
0.35.0
- enh: support new configuration keys for online polygon filtering
in the form of "[online_filter]: area_um,deform polygon points" (#134)
- enh: new helper functions in dclab.definitions: get_config_value_type,
get_config_value_func, and config_key_exists
- ref: issue a warning when the data type of a configuration key is
not correct
0.34.6
- fix: make get_job_info and get_command_log available in cli submodule
- tests: add documentation for test data (#119)
0.34.5
- fix: cast data to float when filling with nan values when checking the
size of ancillary features
- fix: dclab-split did not work when nptdms is not installed
- tests: skip tests that require nptdms, requests, or rpy2 if those
- enh: add check to warn users from setting None-type config values
packaages are not installed
- tests: add test for non-scalar plugin feature
- ci: run tests with different setup.py-extras installed
0.34.4
- fix: use temporary file names during CLI operations
0.34.3
- fix: workaround for when rpy2 finds R_HOME, but R doesn't
0.34.2
- fix: slicing of non-scalar features for hierarchy children (#128)
- fix: passing a directory to the CLI tdms2rtdc did not work
- enh: support slicing of non-scalar features for DCOR data (#132)
- enh: support slicing of the contour feature for HDF5 data (#132)
- enh: support slicing of LazyContour for contour AncillaryFeature (#132)
- enh: minor optimizations of the DCOR format
- ref: raise NotImplementedError for futile attempts to slice the image,
contour, or mask features for .tdms data (#132)
- ref: cleanup CLI code
- tests: increase coverage of CLI (#116)
0.34.1
- enh: introduce user-defined "user" configuration section (#125)
- scripts: refactor fem2lutiso_std.py and fem2iso_volume.py so that
model-specific post-processing is done in hooks named after the FEM
identifier, e.g. fem_hooks/LE-2D-FEM-19.py (#90)
- docs: new LUT data version 10.6084/m9.figshare.12155064.v3
- ref: minor code cleanup
0.34.0
- feat: introduce user-defined plugin features (#105)
- fix: dclab-verify-dataset now also prints other errors it encounters
- fix: installing the "lmer" R package failed, because "statmod" was not
installed
- fix: correct data types for "fluorescence:sample rate",
"imaging: roi position x", and "imaging: roi position y" (#124)
- enh: support new .rtdc attribute "online_contour:bg empty" which is
True when the online background image is computed only from frames
that do not contain any events (#124)
- enh: `AncillaryFeature` now populates other ancillary features when they
share the same method (#104)
- enh: dclab-verify-dataset now returns a non-zero exit code if there
were errors, alerts, or violations (#120)
- ref: streamlined dataset check function for missing meta data keys
0.33.3
- fix: add "chip identifier" to "setup" configuration section and
make it optional during dataset checks (#109)
- fix: ignore empty metadata strings (partly #109); removed the
`check_metadata_empty_string` check function because it does
not apply anymore
0.33.2
- fix: some datasets with unknown feature names could not be
opened (AssertionError regression in 0.33.1)
- fix: workaround for sporadic JSONDecodeError when accessing DCOR
- ref: cleanup cli.py
- ref: cleanup util.py and deprecate `hash_class` argument in
`hashfile`
0.33.1
- fix: add dataset check for wrong medium written to .rtdc file
by Shape-In
- fix: filters were ignored when exporting trace data to hdf5 (#112)
- enh: allow to set ICue identifier for integrity checking
- ref: code cleanup in export.py
0.33.0
- feat: introduce user-defined temporary features (point 2 in #98)
- fix: catch errors for integrity checks on non-raw datasets (#102)
- fix: add check for negative fluorescence values (#101)
- enh: add metadata keys for baseline offset (#107)
- setup: remove deprecated setup.py test
0.32.5
- fix: add check for zero-flow rate in dclab-verify-dataset
- setup: added new file CREDITS for docs and only use maintainer
in setup.py
- docs: add autodoc to constant variables (#94)
- docs: add "features_innate" and "features_loaded" to scripting
goodies section
- ref: cleanup of RTDCBase class
- ref: int/bool deprecation warnings in numpy 1.20.0 (#93)
- tests: test for area_cvx as a float, consistent with output from
Shape-In (#96)
0.32.4
- fix: TypeError when registering emodulus LUT (#91)
- ref: minor cleanup
0.32.3
- build: use oldest-supported-numpy in pyproject.toml
0.32.2
- fix: export trace data in chunks to avoid out-of-memory
errors when compressing large files
- ref: introduce CHUNK_SIZE in write_hdf5.py and use it when
exporting to .rtdc
0.32.1
- enh: dclab-compress now by default does not compress any input
files that are already fully compressed (fully compressed means
that all HDF5 datasets are compressed somehow); to get the old
behavior back (compress in any case, use the "force" keyword
argument)
0.32.0
- feat: allow to register external look-up tables for Young's
modulus computation (#88)
- ref: restructure look-up table file handling
- ref: deprecated [calculation]: "emodulus model" metadata key in favor
of the more descriptive "emodulus lut" key.
- ref: the "method" argument in the context of isoelasticity lines
is deprecated in favor of the "lut_identifier" argument
0.31.5
- fix: writing "filtering" and "calculation" metadata sections
to .rtdc files should not be allowed
0.31.4
- ci: fix rtd builds
- ci: fix PyPI releases
0.31.3
- ci: migrate to GitHub Actions
0.31.2
- enh: add soft type check (assertion) for "emodulus medium"
key in ancillary features (#86)
- fix: make sure that strings are not written as bytes in
hdf5 files
0.31.1
- enh: add boolean "model converged" key to return dictionary of
`Rlme4.fit` (#85)
0.31.0
- feat: implement (generalized) linear mixed-effects models
via a wrapper around R/lme4 using rpy2 (install with extra
"lme4")
0.30.1
- fix: `new_dataset` attempts to load DCOR dataset when
given a non-existent path as a string (#81)
0.30.0
- BREAKING CHANGE: drop support for Python 2 (#34)
- feat: new machine learning (ML) submodule `dclab.ml`
- feat: implement ML model file format .modc (#78)
- feat: add tensorflow helper functions for RT-DC data
- setup: bump numpy>=1.17.0
- ref: minor improvements of code readability
- tests: set global temp dir and remove it after testing
0.29.1
- enh: lift restrictions on valid options for [setup]:medium
(can now be any arbitrary string)
0.29.0
- feat: support the "image_bg" feature which contains the
rolling mean background image computed by Shape-In
0.28.0
- feat: new CLI command dclab-split to split a large dataset into
multiple smaller datasets
0.27.11
- fix: do not cache hierarchy child feature values; this might lead
to intransparent situations where a child has different features
than its parent (you cannot always rely on the user to call
`apply_filter`)
- fix: hierarchy child configuration section "calculation" was not
updated with the hierarchy parent values
- docs: add example for loading data from DCOR and computing
the Young's modulus
0.27.10
- fix: support unicode characters when writing HDF5 in Python2
0.27.9
- docs: add artwork
- fix: support unicode characters in sample names in Python2
0.27.8
- docs: add more information on emodulus computation
- docs: add script for visualizing emodulus LUTs
0.27.7
- ref: replace deprecated .tostring() with .tobytes()
0.27.6
- fix: video seek issue workaround also for the first 100 frames
- cli: also skip the final event in tdms2rtdc if the image is empty
- cli: renamed kwarg `--include-initial-empty-image`
to `include-empty-boundary-images`
- enh: improve detection and recovery of missing images for fmt_tdms
0.27.5
- maintenance build
0.27.4
- maintenance build
0.27.3
- fix: ignore ResourceWarning due to unknown _io.BufferedReader
in third-party software when converting .tdms to .rtdc
0.27.2
- maintenance build
0.27.1
- setup: bump imageio to 2.8.0 for Python>=3.4
- ref: removed NoImageWarning during export (warning is already
issued by fmt_tdms.event_image)
0.27.0
- feat: introduce new feature names `ml_score_???` where `?`
can be a digit or a lower-case letter of the alphabet (#77)
- feat: introduce new functions `dclab.definitions.feature_exists`
and `dclab.definitions.scalar_feature_exists` for checking the
existence of features (including the `ml_score_???` features
which are not in `dclab.definitions.feature_names`)
- feat: introduce ancillary feature `ml_class` which is
defined by the `ml_score_???` features
- enh: fmt_dict automatically converts scalar features to arrays
- ref: replace check for `dclab.definitions.feature_names`
by `dclab.definitions.feature_exists` where applicable
- ref: replace access of `dclab.definitions.feature_name2label`
by `dclab.definitions.get_feature_label` where applicable
- ref: do not automatically fill up all the box filtering ranges
in `RTDCBase.config["filtering"]` with zeros; raise ValueError
if user forgets to set both ranges
- docs: major revision (promote Shape-Out 2 and DCOR)
0.26.2
- fix: `kde_methods.bin_num_doane` now uses 5 as default if it
ecounters nan or zero-division
- docs: updates related to Young's modulus computation
0.26.1
- enh: cache more online data in fmt_dcor
- enh: add `dclab.warn.PipelineWarning` which is used as a parent
class for warnings that a user might be interested in
- fix: temperature warnings during emodulus computation revealed
only the lower temperature limit of the data
0.26.0
- feat: implement volume-deformation isoelasticity lines (#70)
- fix: specifying an external LUT as ndarray did not work
- scripts: finish 'fem2iso_volume.py' for extracting volume-
deformation isoelasticity lines
- scripts: add 'pixelation_correction.py' for visualizing
pixelation effects on area_um, volume, and emodulus
- ref: renamed isoelasticity line text files
0.25.0
- fix: appending data to an hdf5 file results in a broken "index"
feature (re-enumeration from 0), if the given dataset contains
the "index_online" feature
- enh: allow to set external LUT files or LUT data when computing
the Young's modulus with the `lut_data` keyword argument in
`dclab.features.emodulus.get_emodulus`.
- ref: refactored `features.emodulus`: New submodules `pxcorr` and
`scale_linear`; `convert` is deprecated in favor of `scale_feature`
0.24.8
- setup: include Python 3.8 builds and remove Python<=3.5 builds
- scripts: renamed 'extract_lut_and_iso.py' to 'fem2lutiso_std.py'
0.24.7
- fix: `ConfigurationDict.update` did not take into account
invalid keys (everything is now done with (__setitem__`)
0.24.6
- maintenance release
0.24.5
- maintenance release
0.24.4
- maintenance release
0.24.3
- fix: `ConfigurationDict.update` did not actually perform the
requested update (does not affect `Configuration.update`)
- enh: also use points_in_polygon from scikit-image to determine
contour levels
0.24.2
- build: import new skimage submodules so that PyInstaller
will find and use them
0.24.1
- enh: improve polygon filter speed by roughly two orders of
magnitude with a cython version taken from scikit-image; there
are only minor differences to the old implementation (top right
point included vs. lower left point included), so this is not
a breaking change (#23)
0.24.0
- data: refurbished LUT for linear elastic spheres provided by
Dominic Mokbel and Lucas Wittwer (based on the FEM simulation
results from https://doi.org/10.6084/m9.figshare.12155064.v2);
compared to the old LUT, there is a relative error in Young's
modulus below 0.1 %, which should not cause any breaking changes
- data: updated isoelasticity lines (better spacing): analytical
data was made available by Christoph Herold, numerical data
was interpolated from the new LUT
- scripts: added 'scripts/extract_lut_and_iso.py' for extracting
Young's modulus LUT and isoelastics from FEM simulation data
provided by Lucas Wittwer; this is now the default method for
extracting new LUTs and isoelastics
- scripts: added 'scripts/fem2rtdc.py' for generating in-silico
.rtdc datasets from FEM simulation data provided by Lucas Wittwer
- fix: dclab-verify-dataset failed when the "logs" group was not
present in HDF5 files
- fix: use predefined chunks when writing HDF5 data to avoid
exploding file sizes when writing one event at a time
- fix: create a deep copy of the metadata dictionary when writing
HDF5 data because it leaked to subsequent calls
- ref: changed the way isoelasticity lines and emodulus LUTs
are stored and loaded (e.g. json metadata and a few more sanity
checks)
0.23.0
- feat: enable emodulus extrapolation for `area_um`/`deform` values
outside of the given LUT.
0.22.7
- enh: dclab-verify-dataset now also checks whether the sheath and
sample flow rates add up to the channel flow rate
- ref: Configuration does not anymore load unknown meta data
keyword arguments, but ignores them. This implies that
dclab-verify-dataset will not anymore check them actively.
Instead, any warning issued when opening a file is added
to the list of cues.
- setup: bump nptdms to 0.23.0
0.22.6
- fix: data export to HDF5 did not work when the "frame rate"
is not given in the configuration
0.22.5
- enh: add checks for valid keys in the Configuration dictionary
of a dataset `RTDCBase().config`; unknown keys will issue
an UnknownConfigurationKeyWarning (#58)
- ref: moved `rtdc_dataset.fmt_hdf5.UnknownKeyWarning` to
`rtdc_dataset.config.UnknownConfigurationKeyWarning`
- ref: renamed `rtdc_dataset.config.CaseInsensitiveDict`
to `rtdc_dataset.config.ConfigurationDict` and added option
to check new keys
0.22.4
- fix: disable computation of Young's modulus for reservoir
measurements (#75)
- enh: new keyword argument `req_func` for `AncillaryFeature` to
define additional logic for checking whether a feature is
available for a given instance of `RTDCBase`.
0.22.3
- enh: add `data` property to ICues (and use it when checking for
compression)
0.22.2
- fix: when computing the contour from the mask image, always use
the longest contour - critical when the mask image contains
artefacts
- fix: minor issue with dclab-verify-dataset when nptdms was not
installed and an exception occured
- enh: dclab-verify-dataset shows some info on data compression
0.22.1
- enh: remember working API Key
- docs: document DCOR format
0.22.0
- feat: implement DCOR client
- enh: improved .rtdc file format detection (with wrong extension)
0.21.2
- enh: dclab-verify-dataset now also checks HDF5 "mask" feature
attributes
- setup: bump h5py to 2.10.0 (need `<object>.attrs.get_id`)
0.21.1
- fix: correct type of HDF5 image attributes for "mask" feature
0.21.0
- feat: implement new CLI dclab-repack
- fix: don't write "logs" group to HDF5 files if there aren't any
- fix: support HDF5 files that have no "logs" group
- docs: fix docstring of dclab-join
0.20.8
- fix: regression where old .tmds data could not be opened if
they did not contain the "area_msd" feature
- fix: convert bytes logs to string in fmt_hdf5
- enh: support len(ds.logs) for fmt_hdf5
- enh: replace "info" by "build" in CLI job info
0.20.7
- fix: ensure file extension is .rtdc in dclab-join
- fix: correct "frame" and "index_online" features when exporting
to hdf5
- enh: allow to set metadata dictionary in dclab.cli.join
0.20.6
- fix: typo in contour check resulted in small tolerance
0.20.5
- fix: be more trustful when it comes to contour data in the tdms
file format; instead of raising errors, issue warnings (#72)
0.20.4
- ref: move integrity checks to new class check.IntegrityChecker
- docs: document remaining dictionaries in dclab.dfn
0.20.3
- docs: fix bad anchors
0.20.2
- ref: using temperature values outside the range for viscosity
computation now issues a warning instead of raising an error;
warnings were added for the CellCarrier buffers
- fix: handle number detection correctly in get_emodulus
0.20.1
- fix: always return an array when computing the KDE
- ref: make accessible static function RTDCBase.get_kde_spacing
0.20.0
- feat: compute elastic modulus from "temp" feature (#51)
- enh: computing isoelastics from datasets can use
[setup]: "temperature" to compute the viscosity/emodulus (#51)
- enh: define new meta data key [setup]: "temperature"
- docs: add an advanced section on Young's modulus computation (#51)
0.19.1
- fix: hierarchy children did not pass `force` argument to
hierarchy parent when `apply_filter` is called
- fix: revert histogram2d "density" argument to "normed" to support
numpy 1.10 (Shape-Out 1)
- fix: implement unambiguous `RTDCBase.__repr__`
0.19.0
- feat: added better contour spacing computation based on percentiles
(dclab.kde_methods.bin_width_percentile)
- feat: add feature "index_online" which may be missing values (#71)
- feat: implement __getstate__ and __setstate__ for polygon filters
- fix: write UTF-8 BOM when exporting to .tsv
- enh: add check whether unique_id exists in PolygonFilter
0.18.0
- fix: correctly handle filtering when features are removed from a
dataset
- ref: move dclab.rtdc_dataset.util to dclab.util
- ref: minor cleanup in computation of viscosity (support lower-case
`medum` values, add `dclab.features.emodulus_viscosity.KNOWN_MEDIA`)
- ref: cleanup dclab.rtdc_dataset.filter (use logical operators,
correctly display nan-warning messages, keep track of polygon
filters, add consistency checks, improve readability)
0.17.1
- maintenance release
0.17.0
- feat: add command line script for compressing HDF5 (.rtdc)
data "dclab-compress"
- enh: record warnings under "/log" for all command line scripts
- enh: set gzip data compression for all command line scripts
0.16.1
- fix: circumvent UnicodeDecodeErrors which occured in frozen macOS
binaries that use dclab
- enh: support subsecond accuracy in the the configuration key
[experiment] time (e.g. "HH:MM:SS.SSS" instead of "HH:MM:SS")