Add result for Snowflake native load for 10GB benchmark dataset

[sunank200]

Description

What is the current behavior?

Current benchmarking script runs fine for 10 GB dataset. Following is the log I got inside GKE

+ cat /tmp/results-2022-08-22T09:01:07.ndjson
{"duration": 76.34764, "memory_full_info": {"rss": 57171968, "vms": 691195904, "shared": 17715200, "text": 0, "lib": 0, "data": 98746368, "dirty": 0, "uss": 58200064, "pss": 58257408, "swap": 0}, "cpu_time": {"user": 2.53, "system": 0.09000000000000002, "children_user": 0.0, "children_system": 0.0, "iowait": 0.05}, "disk_usage": 274432, "io_counters": [{"read_count": 3581, "write_count": 1178, "read_bytes": 0, "write_bytes": 3051520, "read_chars": 14738885, "write_chars": 2992841}], "dag_id": "load_file_ten_kb_into_snowflake", "execution_date": "2022-08-22 09:01:18.341592+00:00", "revision": "bc58830", "chunk_size": 1000000}
{"duration": 18.347143, "memory_full_info": {"rss": 46039040, "vms": 88358912, "shared": 7073792, "text": 0, "lib": 0, "data": 38658048, "dirty": 0, "uss": 45768704, "pss": 45806592, "swap": 0}, "cpu_time": {"user": 2.5300000000000002, "system": 0.10000000000000003, "children_user": 0.0, "children_system": 0.01, "iowait": 0.04}, "disk_usage": 12288, "io_counters": [{"read_count": 3674, "write_count": 1169, "read_bytes": 0, "write_bytes": 3026944, "read_chars": 14804933, "write_chars": 2966306}], "dag_id": "load_file_hundred_kb_into_snowflake", "execution_date": "2022-08-22 09:02:43.952741+00:00", "revision": "bc58830", "chunk_size": 1000000}
{"duration": 17.899456, "memory_full_info": {"rss": 46796800, "vms": 99713024, "shared": 7159808, "text": 0, "lib": 0, "data": 50012160, "dirty": 0, "uss": 46239744, "pss": 46354432, "swap": 0}, "cpu_time": {"user": 2.54, "system": 0.13, "children_user": 0.0, "children_system": 0.0, "iowait": 0.04}, "disk_usage": 24576, "io_counters": [{"read_count": 7387, "write_count": 1167, "read_bytes": 0, "write_bytes": 3026944, "read_chars": 25012878, "write_chars": 2965272}], "dag_id": "load_file_ten_mb_into_snowflake", "execution_date": "2022-08-22 09:03:12.700308+00:00", "revision": "bc58830", "chunk_size": 1000000}
{"duration": 80.172011, "memory_full_info": {"rss": 47734784, "vms": 90038272, "shared": 7430144, "text": 0, "lib": 0, "data": 40337408, "dirty": 0, "uss": 48058368, "pss": 48061440, "swap": 0}, "cpu_time": {"user": 8.76, "system": 5.59, "children_user": 0.0, "children_system": 0.01, "iowait": 0.02}, "disk_usage": 49152, "io_counters": [{"read_count": 264496, "write_count": 1168, "read_bytes": 0, "write_bytes": 3026944, "read_chars": 1061696749, "write_chars": 2966214}], "dag_id": "load_file_one_gb_into_snowflake", "execution_date": "2022-08-22 09:03:39.289029+00:00", "revision": "bc58830", "chunk_size": 1000000}
{"duration": 348.871686, "memory_full_info": {"rss": 53755904, "vms": 95588352, "shared": 7655424, "text": 0, "lib": 0, "data": 45887488, "dirty": 0, "uss": 53551104, "pss": 53575680, "swap": 0}, "cpu_time": {"user": 18.84, "system": 1.52, "children_user": 0.0, "children_system": 0.0, "iowait": 0.03}, "disk_usage": 184320, "io_counters": [{"read_count": 82125, "write_count": 6593, "read_bytes": 0, "write_bytes": 18882560, "read_chars": 334213220, "write_chars": 19131155}], "dag_id": "load_file_five_gb_into_snowflake", "execution_date": "2022-08-22 09:05:08.624897+00:00", "revision": "bc58830", "chunk_size": 1000000}
{"duration": 493.471326, "memory_full_info": {"rss": 53747712, "vms": 96100352, "shared": 7077888, "text": 0, "lib": 0, "data": 46399488, "dirty": 0, "uss": 53477376, "pss": 53534720, "swap": 0}, "cpu_time": {"user": 34.05, "system": 2.6, "children_user": 0.0, "children_system": 0.01, "iowait": 0.05}, "disk_usage": 258048, "io_counters": [{"read_count": 92837, "write_count": 11881, "read_bytes": 0, "write_bytes": 34344960, "read_chars": 384505742, "write_chars": 34896552}], "dag_id": "load_file_ten_gb_into_snowflake", "execution_date": "2022-08-22 09:11:07.453323+00:00", "revision": "bc58830", "chunk_size": 1000000}
 is not defined
+ [[ -z '' ]]
+ echo ' is not defined'
+ gsutil cp /tmp/results-2022-08-22T09:01:07.ndjson gs://dag-authoring/benchmark/results/
Copying file:///tmp/results-2022-08-22T09:01:07.ndjson [Content-Type=application/octet-stream]...
/ [1 files][  3.7 KiB/  3.7 KiB]                                                
Operation completed over 1 objects/3.7 KiB.                                      
+ command -v peekprof
+ IFS=' '
+ read -r dataset
+ IFS=' '
+ read -r database
+ jq -r '.datasets[] | [.name] | @tsv' /opt/app/tests/benchmark/config.json
+ IFS=' '
+ read -r dataset
parse error: Expected another array element at line 9, column 3
+ IFS=' '
+ read -r database
+ jq -r '.datasets[] | [.name] | @tsv' /opt/app/tests/benchmark/config.json
+ IFS=' '
+ read -r dataset
parse error: Expected another array element at line 9, column 3
+ IFS=' '
+ read -r database

echo Benchmark test completed!
+ echo Benchmark test 'completed!'
Benchmark test completed!

Not able to replicate the error we got earlier with the dataset. It just rans fine. Ran the following:

1. make run_job
2. Got the pod using kubectl get pods -n benchmark
3. Got the logs from kubectl logs -n benchmark benchmark-b9npk-pkrvq -f

closes: #582

What is the new behavior?

• Added the result for Snowflake native load for 10GB benchmark dataset

Does this introduce a breaking change?

No

Checklist

• Created tests which fail without the change (if possible)
• Extended the README / documentation, if necessary

[codecov]

Codecov Report

Merging #712 (330ad4b) into main (f00c4b5) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #712   +/-   ##
=======================================
  Coverage   93.45%   93.45%           
=======================================
  Files          42       42           
  Lines        1726     1726           
  Branches      213      213           
=======================================
  Hits         1613     1613           
  Misses         91       91           
  Partials       22       22           

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.