Bonnie version merge (#72)

* Fixed SDC loading to work with newer DRC measures. And cherry-pick work from master. (#39) * Fixed SDC loading to work with newer DRC measures. - Fixed issues with most data criteria getting thrown out. * Bring over dependabot nokogiri update and the simplexml_parser removal from #30. [Security] Update nokogiri requirement from ~> 1.8.5 to >= 1.8.5, < 1.11.0 Updates the requirements on [nokogiri](https://github.com/sparklemotion/nokogiri) to permit the latest version. - [Release notes](https://github.com/sparklemotion/nokogiri/releases) - [Changelog](https://github.com/sparklemotion/nokogiri/blob/master/CHANGELOG.md) - [Commits](sparklemotion/nokogiri@v1.8.5...v1.10.3) * Add the hqmf identifier to a statement reference (#25) * Port ratio/proportional cv fix from hds and add tests (#48) * codeListId and hqmfOid are both needed for sdc uniqueness * Add descriptive error message if model cannot be found * 2019 standards update (#63) * 2019 standards update entry point fix (#54) * fixed gem entry point file to be named properly * fix issue with loading api uploaded files (#55) * [Security] Bump nokogiri from 1.10.3 to 1.10.4 * Bump cqm-models version to 3.0.0 * [Security] Update rubyzip requirement from ~> 1.2.2 to >= 1.2.2, < 2.1.0 (#67) * [Security] Update rubyzip requirement from ~> 1.2.2 to >= 1.2.2, < 2.1.0 Updates the requirements on [rubyzip](https://github.com/rubyzip/rubyzip) to permit the latest version. - [Release notes](https://github.com/rubyzip/rubyzip/releases) - [Changelog](https://github.com/rubyzip/rubyzip/blob/master/Changelog.md) - [Commits](rubyzip/rubyzip@v1.2.3...v2.0.0) Signed-off-by: dependabot-preview[bot] <support@dependabot.com> * Updated rubyzip dependency to be less than version 2.x, which requires ruby 2.4 * BONNIE-593 Bonnie Unresponsive Message and Error Loading Measure Packages * BONNIE-593 Bonnie Unresponsive Message and Error Loading Measure Packages * BONNIE-593 Bonnie Unresponsive Message and Error Loading Measure Packages * Bonnie-593(ONCJira) test case fix * BONNIE-593 Bonnie Unresponsive Message and Error Loading Measure Packages Fixed vulnerability: sparklemotion/nokogiri#1943 * BONNIE-587 Error loading VSAC value sets(ONC jira id) * Updated version of bonnie_version cqm-parser branch (#71) * Updated version of bonnie_version cqm-parser branch * Updated cqm-parser (binnie_viersion branch) * Remove unnecessary fixtures and re-include test_5_4_CQL_measure_with_drc Co-authored-by: hossenlopp <hossenlopp@mitre.org> Co-authored-by: Luke Osborne <luke.w.osborne@gmail.com> Co-authored-by: dczulada <dczulada@users.noreply.github.com> Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com> Co-authored-by: Ashok Dongare <ashok.dongare@semanticbits.com>
projecttacoma · Feb 6, 2020 · fd4d124 · fd4d124
1 parent 7f1a0de
commit fd4d124
Show file tree

Hide file tree

Showing 17 changed files with 1,120 additions and 12,110 deletions.
diff --git a/.rubocop_todo.yml b/.rubocop_todo.yml
@@ -579,7 +579,7 @@ Naming/AccessorMethodName:
 # AllowedAcronyms: CLI, DSL, ACL, API, ASCII, CPU, CSS, DNS, EOF, GUID, HTML, HTTP, HTTPS, ID, IP, JSON, LHS, QPS, RAM, RHS, RPC, SLA, SMTP, SQL, SSH, TCP, TLS, TTL, UDP, UI, UID, UUID, URI, URL, UTF8, VM, XML, XMPP, XSRF, XSS
 Naming/FileName:
   Exclude:
-    - 'lib/hqmf-parser.rb'
+    - 'lib/cqm-parsers.rb'
 
 Naming/MethodName:
   Exclude:

diff --git a/Gemfile b/Gemfile
@@ -2,9 +2,9 @@ source 'https://rubygems.org'
 
 gemspec :development_group => :test
 
-gem 'mongoid', '~> 6.4.2'
+gem 'mongoid', '~> 5.0.0'
 
-gem 'cqm-models', '~> 2.0.0'
+# gem 'cqm-models', '~> 3.0.0'
 # gem 'cqm-models', git: 'https://github.com/projecttacoma/cqm-models.git', branch: 'master'
 # gem 'cqm-models', :path => '../cqm-models'
 

diff --git a/cqm-parsers.gemspec b/cqm-parsers.gemspec
@@ -9,26 +9,27 @@ Gem::Specification.new do |s|
   s.authors = ["The MITRE Corporation"]
   s.license = 'Apache-2.0'
 
-  s.version = '2.0.1'
+  s.version = '0.2.1.1'
 
+  s.add_dependency 'cqm-models', '~> 3.0.0'
   s.add_dependency 'mustache'
-  s.add_dependency 'erubis', '~> 2.7'
-  s.add_dependency 'mongoid', '~> 6.4'
-  s.add_dependency 'mongoid-tree', '~> 2.1'
-  s.add_dependency 'activesupport', '~> 5.0'
+  s.add_dependency 'erubis', '~> 2.7.0'
+  s.add_dependency 'mongoid', '~> 5.0.0'
+  s.add_dependency 'mongoid-tree', '~> 2.0.0'
+  s.add_dependency 'activesupport', '~> 4.2.0'
 
-  s.add_dependency 'protected_attributes_continued', '~> 1.4'
-  s.add_dependency 'uuid', '~> 2.3'
+  s.add_dependency 'protected_attributes', '~> 1.0.5'
+  s.add_dependency 'uuid', '~> 2.3.7'
   s.add_dependency 'builder', '~> 3.1'
-  s.add_dependency 'nokogiri', '~> 1.10'
-  s.add_dependency 'highline', "~> 1.7"
+  s.add_dependency 'nokogiri', '>= 1.8.5', '< 1.11.0'
+  s.add_dependency 'highline', "~> 1.7.0"
 
-  s.add_dependency 'rubyzip', '~> 1.2'
-  s.add_dependency 'typhoeus', '~> 1.3'
+  s.add_dependency 'rubyzip', '~> 1.3'
+  s.add_dependency 'typhoeus'
   s.add_dependency 'zip-zip', '~> 0.3'
 
-  s.add_dependency 'log4r', '~> 1.1'
-  s.add_dependency 'memoist', '~> 0.9'
+  s.add_dependency 'log4r', '~> 1.1.10'
+  s.add_dependency 'memoist', '~> 0.9.1'
 
   s.files = Dir.glob('lib/**/*.rb') + Dir.glob('lib/**/*.json') + Dir.glob('lib/**/*.mustache') + Dir.glob('lib/**/*.rake') + ["Gemfile", "README.md", "Rakefile"]
 end
diff --git a/lib/hqmf-parser.rb → lib/cqm-parsers.rb b/lib/hqmf-parser.rb → lib/cqm-parsers.rb
diff --git a/lib/hqmf-parser/cql/document_helpers/doc_population_helper.rb b/lib/hqmf-parser/cql/document_helpers/doc_population_helper.rb
@@ -46,9 +46,9 @@ def extract_observations
           measure_population_id = entry.at_xpath("*/cda:measureObservationDefinition/cda:component/cda:criteriaReference/cda:id").attributes['root'].value
           # Get the name of the parameter to the  observation function within the measurePopulationCriteria section
 
-          measure_population_name = entry.at_xpath("*/cda:measureObservationDefinition/cda:component/cda:criteriaReference/cda:id").attributes['extension'].value
-          # Get the name of the parameter to the observation function within the relevant population criteria section
-          cql_define_function[:parameter] = @doc.at_xpath("cda:QualityMeasureDocument/cda:component/cda:populationCriteriaSection/cda:component/cda:#{measure_population_name}Criteria/cda:id[@root = \"#{measure_population_id}\"]/../cda:precondition/cda:criteriaReference/cda:id").attributes['extension'].value.match('\\"([A-Za-z0-9 ]+)\\"')[1]
+          measure_population_name = entry.at_xpath("*/cda:measureObservationDefinition/cda:component/cda:criteriaReference/cda:id").attributes['extension'].value 
+          criteria_reference_id = @doc.at_xpath("cda:QualityMeasureDocument/cda:component/cda:populationCriteriaSection/cda:component/cda:#{measure_population_name}Criteria/cda:id[@root = \"#{measure_population_id}\"]/../cda:precondition/cda:criteriaReference/cda:id")
+          cql_define_function[:parameter] = criteria_reference_id.attributes['extension'].value.match('\\"([A-Za-z0-9 ]+)\\"')[1]
 
           @observations << cql_define_function
         end

diff --git a/lib/measure-loader/mat_measure_files.rb b/lib/measure-loader/mat_measure_files.rb
@@ -53,7 +53,7 @@ class << self
 
       def unzip_measure_zip_into_hash(zip_file)
         folders = Hash.new { |h, k| h[k] = {files: []} }
-        Zip::File.open zip_file do |file|
+        Zip::File.open zip_file.path do |file|
           file.each do |f|
             pn = Pathname(f.name)
             next if '__MACOSX'.in? pn.each_filename  # ignore anything in a __MACOSX folder

diff --git a/lib/measure-loader/source_data_criteria_loader.rb b/lib/measure-loader/source_data_criteria_loader.rb
@@ -26,7 +26,9 @@ def modelize_data_criteria_entry(entry)
                         extract_fields_from_single_code_reference_data_criteria(criteria)
                       end
       hqmf_template_oid = criteria.at_css('templateId/item')['root']
-      model = QDM::ModelFinder.by_hqmf_oid(hqmf_template_oid).new(model_fields)
+      model = QDM::ModelFinder.by_hqmf_oid(hqmf_template_oid)
+      raise "No datatype found for oid #{hqmf_template_oid}. Verify the QDM version of the measure package is correct." if model.nil?
+      model = model.new(model_fields)
       model.description = model.qdmTitle + ': ' + model.description
       return model
     end
@@ -42,16 +44,24 @@ def extract_fields_from_standard_data_criteria(criteria, entry)
     end
 
     def extract_fields_from_single_code_reference_data_criteria(criteria)
-      single_code_reference = criteria.css('value[codeSystem][code]').last || criteria.css('code[codeSystem][code]').last
+      single_code_reference = criteria.at_css('value[codeSystem][code]') || criteria.at_css('code[codeSystem][code]')
       system_id = "#{single_code_reference['codeSystem']}_#{single_code_reference['codeSystemVersion']}".to_sym
-      concept = @single_code_concepts[system_id][single_code_reference['code'].to_sym]
+      concept = @single_code_concepts[system_id][single_code_reference['code'].to_sym] || get_concept_from_participation(criteria.at_css('participation'))
       value_set = concept._parent
       return {
         description: concept.display_name,
         codeListId: value_set.oid
       }
     end
 
+    # If QDM datatype template in MAT has includeSubTemplate, code gets nested into participation
+    # this method gets the codes from participation and form the code concept
+    def get_concept_from_participation(participation)
+      code_element = participation.at_css('code')
+      system_id = "#{code_element['codeSystem']}_#{code_element['codeSystemVersion']}".to_sym
+      @single_code_concepts[system_id][code_element['code'].to_sym]
+    end
+
     def map_single_code_concepts(value_sets_from_single_code_references)
       single_code_concepts = {}
       value_sets_from_single_code_references.flat_map(&:concepts).each do |concept|

diff --git a/lib/tasks/hqmf.rake b/lib/tasks/hqmf.rake
@@ -1,7 +1,7 @@
 require 'pathname'
 require 'fileutils'
 require 'json'
-require 'hqmf-parser'
+require 'cqm-parsers'
 
 namespace :hqmf do
 

diff --git a/lib/util/vsac_api.rb b/lib/util/vsac_api.rb
@@ -138,7 +138,7 @@ def get_program_names
       # Gets the details for a program. This may be used without credentials.
       #
       # Optional parameter program is the program to request from the API. If it is not provided it will look for
-      # a :program in the config passed in during construction. If there is no :program in the config it will use 
+      # a :program in the config passed in during construction. If there is no :program in the config it will use
       # the DEFAULT_PROGRAM constant for the program.
       #
       # Returns the JSON parsed response for program details.
@@ -159,7 +159,7 @@ def get_program_details(program = nil)
       #   }
       #
       # Optional parameter program is the program to request from the API. If it is not provided it will look for
-      # a :program in the config passed in during construction. If there is no :program in the config it will use 
+      # a :program in the config passed in during construction. If there is no :program in the config it will use
       # the DEFAULT_PROGRAM constant for the program.
       #
       # Returns the name of the latest profile for the given program.
@@ -242,8 +242,8 @@ def process_and_validate_vsac_response(vs_response, expected_oid)
       def get_multiple_valueset_raw_responses(needed_value_sets)
         service_tickets = get_service_tickets(needed_value_sets.size)
 
-        hydra = Typhoeus::Hydra.new # Hydra executes multiple HTTP requests at once
-        requests = needed_value_sets.map do |n| 
+        hydra = Typhoeus::Hydra.new(max_concurrency: 1) # Hydra executes multiple HTTP requests at once
+        requests = needed_value_sets.map do |n|
           request = create_valueset_request(n[:value_set][:oid], service_tickets.pop, n[:vs_vsac_options])
           hydra.queue(request)
           request
@@ -258,7 +258,7 @@ def get_multiple_valueset_raw_responses(needed_value_sets)
       def get_service_tickets(amount)
         raise VSACNoCredentialsError.new unless @ticket_granting_ticket
         raise VSACTicketExpiredError.new if Time.now > @ticket_granting_ticket[:expires]
-        
+
         hydra = Typhoeus::Hydra.new # Hydra executes multiple HTTP requests at once
         requests = amount.times.map do
           request = create_service_ticket_request
@@ -299,15 +299,15 @@ def create_valueset_request(oid, ticket, options = {})
 
       # Create a typheous request for a service ticket (this must be executed later)
       def create_service_ticket_request
-        return Typhoeus::Request.new("#{@config[:auth_url]}/Ticket/#{@ticket_granting_ticket[:ticket]}", 
+        return Typhoeus::Request.new("#{@config[:auth_url]}/Ticket/#{@ticket_granting_ticket[:ticket]}",
                                      method: :post,
                                      params: { service: TICKET_SERVICE_PARAM})
       end
 
       # Use your username and password to retrive a ticket granting ticket from VSAC
       def get_ticket_granting_ticket(username, password)
         response = Typhoeus.post(
-          "#{@config[:auth_url]}/Ticket", 
+          "#{@config[:auth_url]}/Ticket",
           # looks like typheous sometimes switches the order of username/password when encoding
           # which vsac cant handle (!?), so encode first
           body: URI.encode_www_form(username: username, password: password)

diff --git a/test/fixtures/measureloading/CMS144_v5_8_Artifacts_20191104.zip b/test/fixtures/measureloading/CMS144_v5_8_Artifacts_20191104.zip
diff --git a/...sureloading/CVmulti_v5_6_Proportional.zip → ...measureloading/CVmulti_v5_6_Artifacts.zip b/...sureloading/CVmulti_v5_6_Proportional.zip → ...measureloading/CVmulti_v5_6_Artifacts.zip
diff --git a/...ures/measureloading/HyperG_v5_6_Ratio.zip → .../measureloading/HyperG_v5_6_Artifacts.zip b/...ures/measureloading/HyperG_v5_6_Ratio.zip → .../measureloading/HyperG_v5_6_Artifacts.zip