Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

1810 fix #1825

Merged
merged 80 commits into from
Feb 21, 2023
Merged
Changes from 1 commit
Commits
Show all changes
80 commits
Select commit Hold shift + click to select a range
c6aba65
Better logging
devclinton Aug 25, 2022
42d3893
Added back changes made to 1810 original branch
emilydriano Aug 25, 2022
b9745a9
Make test platform compatible with id generators
devclinton Aug 30, 2022
a1ae14c
updated platform in test_item_sequence to 'Test' and added tests base…
emilydriano Aug 31, 2022
0908d9d
* Fix name of dummy suite method
devclinton Aug 31, 2022
ea96c12
Merge remote-tracking branch 'origin/1810-fix' into 1810-fix
devclinton Aug 31, 2022
d2925c5
* Add configuration caching to item sequence
devclinton Aug 31, 2022
33b50ef
Add cache clearing to tests
devclinton Aug 31, 2022
073df66
Fix tests
devclinton Aug 31, 2022
e3ac2b6
Update slurm tests to be UUID agnostic
devclinton Aug 31, 2022
cee87fd
Make test valid
devclinton Aug 31, 2022
c0f6eed
Mark test serial to prevent being ran with other tests because of con…
devclinton Aug 31, 2022
6b8ee82
Rest id on close
devclinton Sep 2, 2022
269f789
Updated documentation for item_sequence plugin, cleaned up test_item_…
emilydriano Sep 2, 2022
c12b778
Added methods to clear configuration before persistence_services and …
emilydriano Sep 6, 2022
ce414fa
Added documentation for new functions related to id generation.
emilydriano Sep 6, 2022
7d42595
Change order to ensure file is locked during write
devclinton Sep 7, 2022
05b6b6d
Add file locking
devclinton Sep 7, 2022
ea70b0b
Relay error if it occurs too much
devclinton Sep 7, 2022
c62bad8
Update tests
devclinton Sep 7, 2022
8000098
* Fix locking on item sequence
devclinton Sep 7, 2022
b20fef3
* Update performance test to add asserts
devclinton Sep 7, 2022
9b2451b
Update nechmark
devclinton Sep 7, 2022
200d457
* Fix linting
devclinton Sep 7, 2022
b2ad98e
* Replace with lru cache to be work with 3.7
devclinton Sep 7, 2022
68c0408
Clear other cache
devclinton Sep 7, 2022
20d67a5
Merge branch 'dev' into 1810-fix
devclinton Sep 8, 2022
7bbccf6
#1930 - New hook spec
devclinton Nov 1, 2022
86362ca
Merge remote-tracking branch 'origin/dev' into 1810-fix
devclinton Nov 1, 2022
fda3fdf
* Fix uid issue
devclinton Nov 2, 2022
a23b40c
* Document uid copy for manual case
devclinton Nov 2, 2022
21ea7c3
Updated default item sequence file to be $HOME/.idmtools/item_sequenc…
emilydriano Nov 3, 2022
40a8c0d
Merge branch '1810-fix' of https://github.com/InstituteforDiseaseMode…
emilydriano Nov 3, 2022
b756f8c
Updated item sequence file load Json error, & added logic to idmtools…
emilydriano Nov 7, 2022
bb621c3
Created test to verify existence and correctness of .bak file (backup…
emilydriano Nov 7, 2022
813af5d
* #1936 - Add constant for IDMTOOLS home path
devclinton Nov 7, 2022
585ac18
* Add post create hook at all different types of objects
devclinton Nov 7, 2022
5f1972a
Fix commit issue
devclinton Nov 7, 2022
118a3d5
* Fix linting
devclinton Nov 7, 2022
92d5059
# 1938 - Add post run hook
devclinton Nov 7, 2022
41650a0
Updated item_sequence plugin from post_create_item to post_run
emilydriano Nov 7, 2022
a0b4e64
Added logic to make slurm fail if directory exists
emilydriano Nov 8, 2022
c238580
Removed duplicate creation of simulation items in test_item_sequence
emilydriano Nov 8, 2022
0a9997d
Updated item_sequence documentation
emilydriano Nov 9, 2022
4d7a49c
- corrected logic to allow user to override exist_ok directory option…
emilydriano Nov 9, 2022
61441a0
Expanded on item_sequence plugin documentation.
emilydriano Nov 11, 2022
53f33eb
added 'performance' pytest mark, applied to test_id_generation
emilydriano Nov 11, 2022
9935564
https://github.com/InstituteforDiseaseModeling/idmtools/issues/1954
devclinton Nov 17, 2022
80a180d
#1954 - Move to strings for ids within core
devclinton Nov 17, 2022
e911817
#1954 - implementation of using str, not uuid
emilydriano Nov 22, 2022
758855d
#1954 - implementation of using str, not uuid
emilydriano Nov 22, 2022
86f05ef
#1954 - restored line in slurm item operations to set uid when conver…
emilydriano Nov 23, 2022
ea79ee0
#1954 - restored line in analyze_manager.py
emilydriano Nov 23, 2022
74d4eee
Merge branch 'dev' into 1810-fix
emilydriano Nov 30, 2022
6efc658
removed redundant Union expressions
emilydriano Nov 30, 2022
611b5f6
fixed linting issues
emilydriano Nov 30, 2022
28cb52f
changed UUID to str in slurm create_sim_dir_map functions
emilydriano Nov 30, 2022
032cdfc
final UUID > str changes, updated sequential_id_example
emilydriano Nov 30, 2022
414524d
excluded performance tests from test-all and test-smoke
emilydriano Nov 30, 2022
1b535d0
added jinja2 (used in item_sequence plugin) to requirements.txt files
emilydriano Nov 30, 2022
4dc93f1
updated pytest-html to 3.1.1
emilydriano Dec 2, 2022
948b128
Including target dir in error message to specify which dir already ex…
emilydriano Dec 5, 2022
197e012
modified experiment result to avoid causing the sims to be iterated u…
emilydriano Dec 12, 2022
71f732a
modified experiment result to avoid causing the sims to be iterated u…
emilydriano Dec 12, 2022
d338808
modified tests to exclude slurm experiment simulation data
emilydriano Dec 12, 2022
1e90f8b
modified create_sim_dir_map test to account for metadata change
emilydriano Dec 13, 2022
600f084
Merge branch 'dev' into 1810-fix
emilydriano Dec 13, 2022
5fdc8b0
changed named 'slurm_experiment' to 'experiment' in test_create_sim_d…
emilydriano Dec 13, 2022
56f26d7
Merge branch '1810-fix' of https://github.com/InstituteforDiseaseMode…
emilydriano Dec 13, 2022
a3899a3
used experiment post create in test execute platform to save simulati…
emilydriano Dec 15, 2022
6b89439
included missing package
emilydriano Dec 15, 2022
8e81982
added backup sequence file process to post-run hook for test-execute …
emilydriano Dec 15, 2022
e3ec545
Merge branch 'dev' into 1810-fix
emilydriano Dec 15, 2022
965a7e6
simplified code with super function
emilydriano Dec 16, 2022
ea7ba8c
added kwargs to post_run_item call
emilydriano Dec 16, 2022
99463d3
Merge branch 'dev' into 1810-fix
emilydriano Feb 9, 2023
14e6c2f
Update item_sequence.py
devclinton Feb 9, 2023
6b9a9aa
Update item_sequence.py
shchen-idmod Feb 9, 2023
32ffb5f
added changes defining 'meta' var in idmtools_platform_slurm platform…
emilydriano Feb 16, 2023
557b1fe
Merge remote-tracking branch 'origin/1810-fix' into 1810-fix
emilydriano Feb 16, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,11 @@
from logging import getLogger, DEBUG
from threading import Lock
from pathlib import Path
from typing import Any, List, Type, Dict, Union, TYPE_CHECKING, Optional
from typing import Any, List, Type, Dict, TYPE_CHECKING, Optional
from idmtools import IdmConfigParser
from idmtools.assets import Asset, AssetCollection
from idmtools.core import EntityStatus, ItemType
from idmtools.core import IDMTOOLS_USER_HOME
from idmtools.entities.experiment import Experiment
from idmtools.entities.iplatform_ops.iplatform_experiment_operations import IPlatformExperimentOperations
from idmtools.utils.file import file_contents_to_generator
Expand All @@ -23,6 +25,7 @@
current_directory = os.path.dirname(os.path.realpath(__file__))
data_path = os.path.abspath(os.path.join(current_directory, "..", "..", "data"))
EXPERIMENTS_LOCK = Lock()
SEQUENCE_FILE_DEFAULT_PATH = IDMTOOLS_USER_HOME.joinpath("itemsequence", "index.json")


class ExperimentDict(dict):
Expand Down Expand Up @@ -231,3 +234,9 @@ def post_run_item(self, experiment: Experiment, **kwargs):
sim_path = Path(exp_path, "simulation_index.json")
with open(sim_path, "w") as f:
json.dump([s.id for s in experiment.simulations], f)

if IdmConfigParser.get_option(None, "id_generator", "uuid").lower() == "item_sequence":
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the almost exactly what super class function of post_run_item does (still missing experiment.post_run(self.platform). so no need to copy code. just need call super function.

sequence_file = Path(
IdmConfigParser.get_option("item_sequence", "sequence_file", SEQUENCE_FILE_DEFAULT_PATH))
sequence_file_bk = f'{sequence_file}.bak'
shutil.copy(sequence_file, sequence_file_bk)