Add flag to select which advertise IP for arrow Flight SQL #2

avantgardnerio · 2022-09-07T14:32:12Z

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

As of recently, we have the ability to proxy flights through the Scheduler, but whether or not the client tries to get the Flight from the Scheduler or the Executor is entirely up to us and the IP address we advertise as having the Flight.

Deployment will be easier if we proxy all the Flights through the scheduler, but some users will want the perceived performance boost of having the client reach out directly to the executors.

Describe the solution you'd like

Since there are two potential modes of operation, we should use command line flags (or environment variables) to determine which mode to use. Currently, the executor has command line arguments called --bind-host and (less aptly named) --external-host. Following this pattern, this issue proposes we add a command line argument to the scheduler called --advertise-ip. If specified, all Flights will be altered to use that address. If not specified, the IPs of the executors will be left intact.

Describe alternatives you've considered

Forcing everyone to proxy through the scheduler
Never proxying through the scheduler

Useful links:

The text was updated successfully, but these errors were encountered:

- Update scheduler_config_spec.toml to include new flag 'advertise_host' - Add advertise_host member variable to SchedulerServer - Add advertise_host argument to new, new_with_policy, new_with_builder, and new_with_state in order to propagate flag - Add None argument to relevant method calls

- Update logic in job_to_fetch_part to use advertise-host flag when it exists - Remove default from advertise_host in scheduler_config_spec.toml - Wrap scheduler_server advertise_host variable in Option - Update scheduler's main.rs to reflect advertise_host being wrapped in Option

- Update scheduler_config_spec.toml to include new flag 'advertise_host' - Add advertise_host member variable to SchedulerServer - Add advertise_host argument to new, new_with_policy, new_with_builder, and new_with_state in order to propagate flag - Add None argument to relevant method calls

- Update logic in job_to_fetch_part to use advertise-host flag when it exists - Remove default from advertise_host in scheduler_config_spec.toml - Wrap scheduler_server advertise_host variable in Option - Update scheduler's main.rs to reflect advertise_host being wrapped in Option

- Update scheduler_config_spec.toml to include new flag 'advertise_host' - Add advertise_host member variable to SchedulerServer - Add advertise_host argument to new, new_with_policy, new_with_builder, and new_with_state in order to propagate flag - Add None argument to relevant method calls

- Update logic in job_to_fetch_part to use advertise-host flag when it exists - Remove default from advertise_host in scheduler_config_spec.toml - Wrap scheduler_server advertise_host variable in Option - Update scheduler's main.rs to reflect advertise_host being wrapped in Option

- Update scheduler_config_spec.toml to include new flag 'advertise_host' - Add advertise_host member variable to SchedulerServer - Add advertise_host argument to new, new_with_policy, new_with_builder, and new_with_state in order to propagate flag - Add None argument to relevant method calls

avantgardnerio added the enhancement New feature or request label Sep 7, 2022

DaltonModlin self-assigned this Sep 27, 2022

DaltonModlin closed this as completed Nov 1, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add flag to select which advertise IP for arrow Flight SQL #2

Add flag to select which advertise IP for arrow Flight SQL #2

avantgardnerio commented Sep 7, 2022

Add flag to select which advertise IP for arrow Flight SQL #2

Add flag to select which advertise IP for arrow Flight SQL #2

Comments

avantgardnerio commented Sep 7, 2022