Merge pull request #684 from mpowered/track_validation_failures

Provide a mechanism to map failures back to input
zdennis · Sep 5, 2020 · 900aace · 900aace
2 parents 87d4598 + 2cdad0b
commit 900aace
Show file tree

Hide file tree

Showing 3 changed files with 27 additions and 16 deletions.
diff --git a/README.markdown b/README.markdown
@@ -250,20 +250,21 @@ Book.import books, recursive: true
 
 ### Options
 
-Key                     | Options               | Default            | Description
------------------------ | --------------------- | ------------------ | -----------
-:validate               | `true`/`false`        | `true`             | Whether or not to run `ActiveRecord` validations (uniqueness skipped). This option will always be true when using `import!`.
-:validate_uniqueness    | `true`/`false`        | `false`            | Whether or not to run uniqueness validations, has potential pitfalls, use with caution (requires `>= v0.27.0`).
-:validate_with_context  | `Symbol`              |`:create`/`:update` | Allows passing an ActiveModel validation context for each model. Default is `:create` for new records and `:update` for existing ones. 
-:on_duplicate_key_ignore| `true`/`false`        | `false`            | Allows skipping records with duplicate keys. See [here](https://github.com/zdennis/activerecord-import/#duplicate-key-ignore) for more details.
-:ignore                 | `true`/`false`        | `false`            | Alias for :on_duplicate_key_ignore.
-:on_duplicate_key_update| :all, `Array`, `Hash` | N/A                | Allows upsert logic to be used. See [here](https://github.com/zdennis/activerecord-import/#duplicate-key-update) for more details.
-:synchronize            | `Array`               | N/A                | An array of ActiveRecord instances. This synchronizes existing instances in memory with updates from the import.
-:timestamps             | `true`/`false`        | `true`             | Enables/disables timestamps on imported records.
-:recursive              | `true`/`false`        | `false`            | Imports has_many/has_one associations (PostgreSQL only).
-:batch_size             | `Integer`             | total # of records | Max number of records to insert per import
-:raise_error            | `true`/`false`        | `false`            | Raises an exception at the first invalid record. This means there will not be a result object returned. The `import!` method is a shortcut for this.
-:all_or_none            | `true`/`false`        | `false`            | Will not import any records if there is a record with validation errors.
+Key                       | Options               | Default            | Description
+------------------------- | --------------------- | ------------------ | -----------
+:validate                 | `true`/`false`        | `true`             | Whether or not to run `ActiveRecord` validations (uniqueness skipped). This option will always be true when using `import!`.
+:validate_uniqueness      | `true`/`false`        | `false`            | Whether or not to run uniqueness validations, has potential pitfalls, use with caution (requires `>= v0.27.0`).
+:validate_with_context    | `Symbol`              |`:create`/`:update` | Allows passing an ActiveModel validation context for each model. Default is `:create` for new records and `:update` for existing ones. 
+:track_validation_failures| `true`/`false`        | `false`            | When this is set to true, `failed_instances` will be an array of arrays, with each inner array having the form `[:index_in_dataset, :object_with_errors]` 
+:on_duplicate_key_ignore  | `true`/`false`        | `false`            | Allows skipping records with duplicate keys. See [here](https://github.com/zdennis/activerecord-import/#duplicate-key-ignore) for more details.
+:ignore                   | `true`/`false`        | `false`            | Alias for :on_duplicate_key_ignore.
+:on_duplicate_key_update  | :all, `Array`, `Hash` | N/A                | Allows upsert logic to be used. See [here](https://github.com/zdennis/activerecord-import/#duplicate-key-update) for more details.
+:synchronize              | `Array`               | N/A                | An array of ActiveRecord instances. This synchronizes existing instances in memory with updates from the import.
+:timestamps               | `true`/`false`        | `true`             | Enables/disables timestamps on imported records.
+:recursive                | `true`/`false`        | `false`            | Imports has_many/has_one associations (PostgreSQL only).
+:batch_size               | `Integer`             | total # of records | Max number of records to insert per import
+:raise_error              | `true`/`false`        | `false`            | Raises an exception at the first invalid record. This means there will not be a result object returned. The `import!` method is a shortcut for this.
+:all_or_none              | `true`/`false`        | `false`            | Will not import any records if there is a record with validation errors.
 
 #### Duplicate Key Ignore
 

diff --git a/lib/activerecord-import/import.rb b/lib/activerecord-import/import.rb
@@ -547,7 +547,7 @@ def bulk_import!(*args)
     alias import! bulk_import! unless ActiveRecord::Base.respond_to? :import!
 
     def import_helper( *args )
-      options = { validate: true, timestamps: true }
+      options = { validate: true, timestamps: true, track_validation_failures: false }
       options.merge!( args.pop ) if args.last.is_a? Hash
       # making sure that current model's primary key is used
       options[:primary_key] = primary_key
@@ -709,7 +709,7 @@ def import_helper( *args )
               array_of_attributes[i] = nil
               failure = model.dup
               failure.errors.send(:initialize_dup, model.errors)
-              failed_instances << failure
+              failed_instances << (options[:track_validation_failures] ? [i, failure] : failure )
             end
             array_of_attributes.compact!
           end

diff --git a/test/import_test.rb b/test/import_test.rb
@@ -252,6 +252,16 @@
         end
       end
 
+      it "should index the failed instances by their poistion in the set if `track_failures` is true" do
+        index_offset = valid_values.length
+        results = Topic.import columns, valid_values + invalid_values, validate: true, track_validation_failures: true
+        assert_equal invalid_values.size, results.failed_instances.size
+        invalid_values.each_with_index do |value_set, index|
+          assert_equal index + index_offset, results.failed_instances[index].first
+          assert_equal value_set.first, results.failed_instances[index].last.title
+        end
+      end
+
       it "should set ids in valid models if adapter supports setting primary key of imported objects" do
         if ActiveRecord::Base.supports_setting_primary_key_of_imported_objects?
           Topic.import (invalid_models + valid_models), validate: true