Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FUTURE: initial support for .npmrc file #23560

Merged
merged 57 commits into from
May 23, 2024
Merged
Show file tree
Hide file tree
Changes from 52 commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
3e511d0
bump crates
bartlomieju Apr 25, 2024
0a7839d
feat: discover .npmrc file
bartlomieju Apr 25, 2024
6abaf3d
get correct URL for info
bartlomieju Apr 25, 2024
deb8367
can auth
bartlomieju Apr 25, 2024
e23e995
bump deno_npm
bartlomieju Apr 25, 2024
797ca2a
change test
bartlomieju Apr 25, 2024
2fefcde
tarballs are downloaded from correct server, but not placed in the co…
bartlomieju Apr 25, 2024
b65c4db
updates to latest changes
bartlomieju Apr 25, 2024
9c40f3c
remove some registry_url vars
bartlomieju Apr 25, 2024
8035357
test is passing
bartlomieju Apr 25, 2024
b2653f6
remove some omre
bartlomieju Apr 25, 2024
61b6407
only two left
bartlomieju Apr 26, 2024
c39adb3
use a candidate list
bartlomieju Apr 26, 2024
105f47b
cowsay still working
bartlomieju Apr 26, 2024
35a8bb1
keep removing
bartlomieju Apr 26, 2024
49d93f0
add TODOs
bartlomieju Apr 26, 2024
69a6fb1
compile test passing again?
bartlomieju Apr 26, 2024
fce7942
Merge branch 'main' into npm_private_registry
bartlomieju Apr 26, 2024
caf1556
bump crates
bartlomieju Apr 26, 2024
896ea1e
tests
bartlomieju Apr 26, 2024
a688f44
fix test
bartlomieju Apr 26, 2024
2376331
Merge branch 'main' into npm_private_registry
bartlomieju Apr 26, 2024
86216ba
Merge remote-tracking branch 'upstream/main' into npm_private_registry
bartlomieju Apr 26, 2024
ab0247a
Merge branch 'main' into npm_private_registry
bartlomieju Apr 27, 2024
24e8510
change server path
bartlomieju Apr 27, 2024
f5d10e7
don't pass auth header if origin changes
bartlomieju Apr 27, 2024
0693bca
Merge branch 'main' into npm_private_registry
bartlomieju Apr 29, 2024
fc556d8
clippy
bartlomieju Apr 29, 2024
d2510c5
update discovery algorithm
bartlomieju Apr 29, 2024
17d7f90
add more tests
bartlomieju Apr 29, 2024
f600398
Merge branch 'main' into npm_private_registry
bartlomieju May 3, 2024
707fb7a
add a second registry
bartlomieju May 3, 2024
afc61d5
lint
bartlomieju May 3, 2024
24e7f85
update the test url
bartlomieju May 3, 2024
b08808d
Merge branch 'main' into npm_private_registry
bartlomieju May 6, 2024
74d794c
Merge branch 'main' into npm_private_registry
bartlomieju May 6, 2024
4186e73
only enable with DENO_FUTURE
bartlomieju May 6, 2024
1077f8e
Revert "only enable with DENO_FUTURE"
bartlomieju May 7, 2024
c026084
Merge branch 'main' into npm_private_registry
bartlomieju May 7, 2024
0d9ca98
Merge branch 'main' into npm_private_registry
bartlomieju May 7, 2024
280a395
try to fix tests
bartlomieju May 7, 2024
0386add
tests passing again
bartlomieju May 7, 2024
0ee94a0
Merge branch 'main' into npm_private_registry
bartlomieju May 8, 2024
7cad76c
Merge branch 'main' into npm_private_registry
bartlomieju May 9, 2024
bc7b91f
fix test
bartlomieju May 9, 2024
1d6bc5e
guard behind DENO_FUTURE
bartlomieju May 9, 2024
e1a9606
Merge branch 'main' into npm_private_registry
bartlomieju May 13, 2024
d0edd15
update lockfile
bartlomieju May 13, 2024
f772ad6
Merge branch 'main' into npm_private_registry
bartlomieju May 14, 2024
29047ee
add tests for deno compile
bartlomieju May 14, 2024
23b54dc
typo
bartlomieju May 14, 2024
7009dd4
Merge branch 'main' into npm_private_registry
bartlomieju May 14, 2024
00cedbd
compute registry dirname in the constructor
bartlomieju May 14, 2024
0944c00
Merge branch 'main' into npm_private_registry
bartlomieju May 22, 2024
30cca3e
add support for _auth
bartlomieju May 22, 2024
88bc520
remove debug output
bartlomieju May 22, 2024
2fd56e7
Merge branch 'main' into npm_private_registry
bartlomieju May 23, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
4 changes: 2 additions & 2 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion cli/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ deno_emit = "=0.40.3"
deno_graph = { version = "=0.75.1", features = ["tokio_executor"] }
deno_lint = { version = "=0.58.4", features = ["docs"] }
deno_lockfile.workspace = true
deno_npm = "=0.20.1"
deno_npm = "=0.20.0"
deno_runtime = { workspace = true, features = ["include_js_files_for_snapshotting"] }
deno_semver = "=0.5.4"
deno_task_shell = "=0.16.1"
Expand Down
98 changes: 98 additions & 0 deletions cli/args/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ use self::package_json::PackageJsonDeps;
use ::import_map::ImportMap;
use deno_ast::SourceMapOption;
use deno_core::resolve_url_or_path;
use deno_npm::npm_rc::NpmRc;
use deno_npm::npm_rc::ResolvedNpmRc;
use deno_npm::resolution::ValidSerializedNpmResolutionSnapshot;
use deno_npm::NpmSystemInfo;
use deno_runtime::deno_tls::RootCertStoreProvider;
Expand Down Expand Up @@ -546,6 +548,83 @@ fn discover_package_json(
Ok(None)
}

/// Discover `.npmrc` file - currently we only support it next to `package.json`
/// or next to `deno.json`.
///
/// In the future we will need to support it in user directory or global directory
/// as per https://docs.npmjs.com/cli/v10/configuring-npm/npmrc#files.
fn discover_npmrc(
maybe_package_json_path: Option<PathBuf>,
maybe_deno_json_path: Option<PathBuf>,
) -> Result<Arc<ResolvedNpmRc>, AnyError> {
if !*DENO_FUTURE {
return Ok(create_default_npmrc());
}

const NPMRC_NAME: &str = ".npmrc";

fn get_env_var(var_name: &str) -> Option<String> {
std::env::var(var_name).ok()
}

fn try_to_read_npmrc(
dir: &Path,
) -> Result<Option<(String, PathBuf)>, AnyError> {
let path = dir.join(NPMRC_NAME);
let maybe_source = match std::fs::read_to_string(&path) {
Ok(source) => Some(source),
Err(err) if err.kind() == std::io::ErrorKind::NotFound => None,
Err(err) => {
bail!("Error loading .npmrc at {}. {:#}", path.display(), err)
}
};

Ok(maybe_source.map(|source| (source, path)))
}

fn try_to_parse_npmrc(
source: String,
path: &Path,
) -> Result<Arc<ResolvedNpmRc>, AnyError> {
let npmrc = NpmRc::parse(&source, &get_env_var).with_context(|| {
format!("Failed to parse .npmrc at {}", path.display())
})?;
let resolved = npmrc
.as_resolved(npm_registry_url())
.context("Failed to resolve .npmrc options")?;
Ok(Arc::new(resolved))
}

if let Some(package_json_path) = maybe_package_json_path {
if let Some(package_json_dir) = package_json_path.parent() {
if let Some((source, path)) = try_to_read_npmrc(package_json_dir)? {
return try_to_parse_npmrc(source, &path);
}
}
}

if let Some(deno_json_path) = maybe_deno_json_path {
if let Some(deno_json_dir) = deno_json_path.parent() {
if let Some((source, path)) = try_to_read_npmrc(deno_json_dir)? {
return try_to_parse_npmrc(source, &path);
}
}
}

log::debug!("No .npmrc file found");
Ok(create_default_npmrc())
}

pub fn create_default_npmrc() -> Arc<ResolvedNpmRc> {
Arc::new(ResolvedNpmRc {
default_config: deno_npm::npm_rc::RegistryConfigWithUrl {
registry_url: npm_registry_url().clone(),
config: Default::default(),
},
scopes: Default::default(),
})
}

struct CliRootCertStoreProvider {
cell: OnceCell<RootCertStore>,
maybe_root_path: Option<PathBuf>,
Expand Down Expand Up @@ -722,6 +801,7 @@ pub struct CliOptions {
maybe_vendor_folder: Option<PathBuf>,
maybe_config_file: Option<ConfigFile>,
maybe_package_json: Option<PackageJson>,
npmrc: Arc<ResolvedNpmRc>,
maybe_lockfile: Option<Arc<Mutex<Lockfile>>>,
overrides: CliOptionOverrides,
maybe_workspace_config: Option<WorkspaceConfig>,
Expand All @@ -736,6 +816,7 @@ impl CliOptions {
maybe_config_file: Option<ConfigFile>,
maybe_lockfile: Option<Arc<Mutex<Lockfile>>>,
maybe_package_json: Option<PackageJson>,
npmrc: Arc<ResolvedNpmRc>,
force_global_cache: bool,
) -> Result<Self, AnyError> {
if let Some(insecure_allowlist) =
Expand Down Expand Up @@ -798,6 +879,7 @@ impl CliOptions {
maybe_config_file,
maybe_lockfile,
maybe_package_json,
npmrc,
maybe_node_modules_folder,
maybe_vendor_folder,
overrides: Default::default(),
Expand Down Expand Up @@ -851,6 +933,16 @@ impl CliOptions {
} else {
maybe_package_json = discover_package_json(&flags, None, &initial_cwd)?;
}
let npmrc = discover_npmrc(
maybe_package_json.as_ref().map(|p| p.path.clone()),
maybe_config_file.as_ref().and_then(|cf| {
if cf.specifier.scheme() == "file" {
Some(cf.specifier.to_file_path().unwrap())
} else {
None
}
}),
)?;

let maybe_lock_file =
lockfile::discover(&flags, maybe_config_file.as_ref())?;
Expand All @@ -860,6 +952,7 @@ impl CliOptions {
maybe_config_file,
maybe_lock_file.map(|l| Arc::new(Mutex::new(l))),
maybe_package_json,
npmrc,
false,
)
}
Expand Down Expand Up @@ -1161,6 +1254,7 @@ impl CliOptions {
maybe_vendor_folder: self.maybe_vendor_folder.clone(),
maybe_config_file: self.maybe_config_file.clone(),
maybe_package_json: self.maybe_package_json.clone(),
npmrc: self.npmrc.clone(),
maybe_lockfile: self.maybe_lockfile.clone(),
maybe_workspace_config: self.maybe_workspace_config.clone(),
overrides: self.overrides.clone(),
Expand Down Expand Up @@ -1292,6 +1386,10 @@ impl CliOptions {
&self.maybe_package_json
}

pub fn npmrc(&self) -> &Arc<ResolvedNpmRc> {
&self.npmrc
}

pub fn maybe_package_json_deps(&self) -> Option<PackageJsonDeps> {
if matches!(
self.flags.subcommand,
Expand Down
2 changes: 1 addition & 1 deletion cli/factory.rs
Original file line number Diff line number Diff line change
Expand Up @@ -447,7 +447,7 @@ impl CliFactory {
self.package_json_deps_provider().clone(),
),
npm_system_info: self.options.npm_system_info(),
npm_registry_url: crate::args::npm_registry_url().to_owned(),
npmrc: self.options.npmrc().clone()
})
}).await
}.boxed_local())
Expand Down
33 changes: 27 additions & 6 deletions cli/http_util.rs
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ use deno_runtime::deno_fetch::reqwest::header::LOCATION;
use deno_runtime::deno_fetch::reqwest::Response;
use deno_runtime::deno_fetch::CreateHttpClientOptions;
use deno_runtime::deno_tls::RootCertStoreProvider;
use reqwest::header::HeaderName;
use reqwest::header::HeaderValue;
use std::collections::HashMap;
use std::sync::Arc;
use std::time::Duration;
Expand Down Expand Up @@ -294,7 +296,7 @@ impl HttpClient {
&self,
url: U,
) -> Result<Vec<u8>, AnyError> {
let maybe_bytes = self.inner_download(url, None).await?;
let maybe_bytes = self.inner_download(url, None, None).await?;
match maybe_bytes {
Some(bytes) => Ok(bytes),
None => Err(custom_error("Http", "Not found.")),
Expand All @@ -304,17 +306,21 @@ impl HttpClient {
pub async fn download_with_progress<U: reqwest::IntoUrl>(
&self,
url: U,
maybe_header: Option<(HeaderName, HeaderValue)>,
progress_guard: &UpdateGuard,
) -> Result<Option<Vec<u8>>, AnyError> {
self.inner_download(url, Some(progress_guard)).await
self
.inner_download(url, maybe_header, Some(progress_guard))
.await
}

async fn inner_download<U: reqwest::IntoUrl>(
&self,
url: U,
maybe_header: Option<(HeaderName, HeaderValue)>,
progress_guard: Option<&UpdateGuard>,
) -> Result<Option<Vec<u8>>, AnyError> {
let response = self.get_redirected_response(url).await?;
let response = self.get_redirected_response(url, maybe_header).await?;

if response.status() == 404 {
return Ok(None);
Expand All @@ -339,15 +345,30 @@ impl HttpClient {
pub async fn get_redirected_response<U: reqwest::IntoUrl>(
&self,
url: U,
mut maybe_header: Option<(HeaderName, HeaderValue)>,
) -> Result<Response, AnyError> {
let mut url = url.into_url()?;
let mut response = self.get_no_redirect(url.clone())?.send().await?;

let mut builder = self.get_no_redirect(url.clone())?;
if let Some((header_name, header_value)) = maybe_header.as_ref() {
builder = builder.header(header_name, header_value);
}
let mut response = builder.send().await?;
let status = response.status();
if status.is_redirection() {
for _ in 0..5 {
let new_url = resolve_redirect_from_response(&url, &response)?;
let new_response =
self.get_no_redirect(new_url.clone())?.send().await?;
let mut builder = self.get_no_redirect(new_url.clone())?;

if new_url.origin() == url.origin() {
if let Some((header_name, header_value)) = maybe_header.as_ref() {
builder = builder.header(header_name, header_value);
}
} else {
maybe_header = None;
}

let new_response = builder.send().await?;
let status = new_response.status();
if status.is_redirection() {
response = new_response;
Expand Down
5 changes: 5 additions & 0 deletions cli/lsp/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ use deno_core::serde_json::json;
use deno_core::serde_json::Value;
use deno_core::ModuleSpecifier;
use deno_lockfile::Lockfile;
use deno_npm::npm_rc::ResolvedNpmRc;
use deno_runtime::deno_node::PackageJson;
use deno_runtime::fs_util::specifier_to_file_path;
use deno_runtime::permissions::PermissionsContainer;
Expand Down Expand Up @@ -1090,6 +1091,7 @@ pub struct ConfigData {
pub vendor_dir: Option<PathBuf>,
pub lockfile: Option<Arc<Mutex<Lockfile>>>,
pub package_json: Option<Arc<PackageJson>>,
pub npmrc: Option<Arc<ResolvedNpmRc>>,
pub import_map: Option<Arc<ImportMap>>,
pub import_map_from_settings: bool,
watched_files: HashMap<ModuleSpecifier, ConfigWatchedFileType>,
Expand Down Expand Up @@ -1274,6 +1276,8 @@ impl ConfigData {

// Load package.json
let mut package_json = None;
// TODO(bartlomieju): support discovering .npmrc
let npmrc = None;
if let Ok(path) = specifier_to_file_path(scope) {
let path = path.join("package.json");
if let Ok(specifier) = ModuleSpecifier::from_file_path(&path) {
Expand Down Expand Up @@ -1429,6 +1433,7 @@ impl ConfigData {
vendor_dir,
lockfile: lockfile.map(Mutex::new).map(Arc::new),
package_json: package_json.map(Arc::new),
npmrc: npmrc.map(Arc::new),
import_map: import_map.map(Arc::new),
import_map_from_settings,
watched_files,
Expand Down
4 changes: 4 additions & 0 deletions cli/lsp/language_server.rs
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,7 @@ use super::tsc::ChangeKind;
use super::tsc::GetCompletionDetailsArgs;
use super::tsc::TsServer;
use super::urls;
use crate::args::create_default_npmrc;
use crate::args::get_root_cert_store;
use crate::args::CaData;
use crate::args::CacheSetting;
Expand Down Expand Up @@ -3272,6 +3273,9 @@ impl Inner {
config_data.and_then(|d| d.config_file.as_deref().cloned()),
config_data.and_then(|d| d.lockfile.clone()),
config_data.and_then(|d| d.package_json.as_deref().cloned()),
config_data
.and_then(|d| d.npmrc.clone())
.unwrap_or_else(create_default_npmrc),
force_global_cache,
)?;

Expand Down
6 changes: 5 additions & 1 deletion cli/lsp/resolver.rs
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
// Copyright 2018-2024 the Deno authors. All rights reserved. MIT license.

use crate::args::create_default_npmrc;
use crate::args::package_json;
use crate::args::CacheSetting;
use crate::cache::FastInsecureHasher;
Expand Down Expand Up @@ -338,7 +339,10 @@ async fn create_npm_resolver(
// do not install while resolving in the lsp—leave that to the cache command
package_json_installer:
CliNpmResolverManagedPackageJsonInstallerOption::NoInstall,
npm_registry_url: crate::args::npm_registry_url().to_owned(),
npmrc: config_data
.npmrc
.clone()
.unwrap_or_else(create_default_npmrc),
npm_system_info: NpmSystemInfo::default(),
})
};
Expand Down