bundler: how can i make --full-index automatic?

[ljharb]

Describe the problem as clearly as you can

At my company, --full-index is used to improve performance when fetching gems from S3 buckets. However, there seems to be no environment variable or config file approach to make this be the default, so engineers have to remember to use it manually.

(https://stackoverflow.com/questions/14263588/how-to-always-run-bundle-install-with-the-full-index-option asks the same question)

Post system independent steps to reproduce the problem

N/A

Which command did you run?

bundle install --full-index

What were you expecting to happen?

N/A

What actually happened?

N/A

If not included with the output of your command, run bundle env and paste the output below

N/A

---

Is there any way to make this (and perhaps other options) configurable in a way that can be stored in a git repo, or as part of an environment?

[colby-swandale]

@ljharb We would welcome a PR to add a config option if this is something you would like to see in Bundler.

Here's an example PR rubygems/bundler#6309

[ljharb]

@colby-swandale thanks - that example PR tho references files and terms that no longer exist, like lib/bundler/settings.rb and disable_platform_warnings. I'll try to figure it out, but if you had a more recent example PR that would be great :-)

[ljharb]

Also, any tips on where would be the best place to read/respect the config value? I was thinking either the CLI (as a lazy_default for --full-index), or in each of the three places Bundler::Fetcher.disable_endpoint is set (but then i'm not sure how i'd figure out when the option was omitted or explicitly provided).

[deivid-rodriguez]

Sorry for chiming in this late.

I'd like to ask some more background for this because for a long time I haven't heard of anyone using this option. When I started contributing to bundler I remember it being recommended on some error messages to "heal" some bad bundle install's but I haven't seen that in such a long time. My understanding was that since the introduction of the new API, this option was no longer necessary.

[ljharb]

@deivid-rodriguez my company's internal bundle installs are quite slow (i believe some gems are fetched from s3 buckets, which is in particular when it's slow), and every engineer is told to use --full-index to speed them up. My intention in filing this issue was to ensure a method to make it fast without providing the option explicitly, ie, by enabling it by default for all engineers via config file or env var.

[deivid-rodriguez]

Hi @ljharb. Yeah, I understand the request 👍.

My concern is that the new API (what's used by default without the --full-index flag) was born to solve a lot of issues with the old API (what --full-index uses). And this has worked fine so far, since as I said before, this is the first time in a very long time that I hear from anyone still using the old API. I mean using it intentionally through the --full-index flag, it is true that under certain conditions, bundler still ends up falling back to using it as a last resort.

As a matter of fact, we have recently started considering stopping using the old API at all in bundler, so that we can eventually phase it out on the server side. A first step in this direction would be to deprecate --full-index.

So I feel adding a configuration to always use --full-index kind of goes against our roadmap, and I'd rather try to figure out why things are working that slow for you with the new API, and try to fix that. Maybe if we can see some logs, or get reproducible simulation of your setup can help us figuring this out.

[deivid-rodriguez]

We haven't received any further feedback, so I'll close this for the moment.

I still would like to figure out what's going on here, so feel free to reopen this and post more information so we can try figure this out!

[ljharb]

@deivid-rodriguez unfortunately i'm unable to share it, but i have a long gist of log output. Since we're using a gem bucket from S3, i don't think we can take advantage of any new APIs :-/

[deivid-rodriguez]

Could you at least share the gist, with any "private stuff" masked or removed or something?

So... You're using a private gem source using gemstash? I think that would make sense since, it seems it doesn't yet support the new Compact Index API. So, that being the case, using --full-index essentially skips all the fallback behaviour of first trying an API that doesn't work, and then trying the API that does work. So it makes sense that it's faster.

Does this make any sense @bronzdoc?

[deivid-rodriguez]

If I'm right, we should move this ticket to gemstash, but for now, just to make sure we don't forget about this, I'm reopening.

[deivid-rodriguez]

@ljharb Can you confirm whether you're using a private gem source through gemstash? I think you should see those 403 responses in the logs if you run bundle install --verbose.

[ljharb]

@deivid-rodriguez we’re not using gemstash, we instead use an s3 bucket as our backend.

[deivid-rodriguez]

@ljharb Ok, thanks. I had a recollection that gemstash had added support for using s3 as the underlying storage for gems, so I figured that would be your setup.

In that case, I have no idea of how setting up an s3 bucket as a backend works with bundler. Could you give me some pointers? How can I setup a similar "toy" application myself? What does the main source in your Gemfile look like?

[bronzdoc]

Gemstash recently added support for a s3 backend but that is not yet released 😬

[deivid-rodriguez]

I'll go ahead and close this since we haven't managed to figure out the root cause of your usage of bundler being faster with --full-index. Happy to continue digging to figure out the culprit, either on the closed issue or reopening it if you provide further feedback.

For now, let's focus on rubygems/gemstash#72, which is the one issue I have found out about from this ticket :)