Replace checked casts with as for performance

[alamb]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
There are places in the code where checked casts are used (e.g. try_into().unwrap()) where the only codepath that leads to this panic are invalid array data, which the code goes through great lengths to avoid (via validate and validate_full).

Describe the solution you'd like

1. replace try_into for such cases with an as cast
2. Run performance benchmarks and see if it improves things (and make a PR if so)

Describe alternatives you've considered
N/A

Additional context
suggested by @tustvold here: https://github.com/apache/arrow-rs/pull/1912/files#r901284237