New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
New Environment model for recipes and profiles #8630
Changes from 34 commits
75ff512
3a585e5
5f323b1
16863a1
3fd4633
de6787e
7f86beb
a96287c
004ab8b
3d318e3
247f98c
997936b
fa0ca3d
e63890e
1a0abdf
1f66a6c
f78169a
b6dcb07
e3a3cf0
1482923
1d42450
935b3d0
15dbf07
4221d32
3798c78
4dbe0c0
e1bf328
c43ed63
3198d92
b84196a
e3a4aa9
2fc9228
392a788
1530a0f
da85931
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -7,7 +7,6 @@ __pycache__/ | |
|
||
# Distribution / packaging | ||
.Python | ||
env/ | ||
build/ | ||
develop-eggs/ | ||
dist/ | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
from conan.tools.env.environment import Environment |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,291 @@ | ||
import fnmatch | ||
import os | ||
import textwrap | ||
from collections import OrderedDict | ||
|
||
from conans.errors import ConanException | ||
from conans.util.files import save | ||
|
||
|
||
class _EnvVarPlaceHolder: | ||
pass | ||
|
||
|
||
class _Sep(str): | ||
pass | ||
|
||
|
||
class _PathSep: | ||
pass | ||
|
||
|
||
def environment_wrap_command(filename, cmd, cwd=None): | ||
assert filename | ||
filenames = [filename] if not isinstance(filename, list) else filename | ||
bats, shs = [], [] | ||
for f in filenames: | ||
full_path = os.path.join(cwd, f) if cwd else f | ||
if os.path.isfile("{}.bat".format(full_path)): | ||
bats.append("{}.bat".format(f)) | ||
elif os.path.isfile("{}.sh".format(full_path)): | ||
shs.append("{}.sh".format(f)) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. should we raise for There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. No, it is injecting the environment file if it is found, but the file might not always be there, for example if there are no environment variables to define, instead of generating a blank file, the file is not being generated. That is what allows to inject previously unknown |
||
if bats and shs: | ||
raise ConanException("Cannot wrap command with different envs, {} - {}".format(bats, shs)) | ||
|
||
if bats: | ||
command = " && ".join(bats) | ||
return "{} && {}".format(command, cmd) | ||
elif shs: | ||
command = " && ".join(". ./{}".format(f) for f in shs) | ||
return "{} && {}".format(command, cmd) | ||
else: | ||
return cmd | ||
|
||
|
||
class Environment: | ||
def __init__(self): | ||
# TODO: Maybe we need to pass conanfile to get the [conf] | ||
# It being ordered allows for Windows case-insensitive composition | ||
self._values = OrderedDict() # {var_name: [] of values, including separators} | ||
|
||
def __bool__(self): | ||
return bool(self._values) | ||
|
||
__nonzero__ = __bool__ | ||
|
||
def __repr__(self): | ||
return repr(self._values) | ||
|
||
def vars(self): | ||
return list(self._values.keys()) | ||
|
||
def value(self, name, placeholder="{name}", pathsep=os.pathsep): | ||
return self._format_value(name, self._values[name], placeholder, pathsep) | ||
|
||
@staticmethod | ||
def _format_value(name, varvalues, placeholder, pathsep): | ||
values = [] | ||
for v in varvalues: | ||
|
||
if v is _EnvVarPlaceHolder: | ||
values.append(placeholder.format(name=name)) | ||
elif v is _PathSep: | ||
values.append(pathsep) | ||
else: | ||
values.append(v) | ||
return "".join(values) | ||
|
||
@staticmethod | ||
def _list_value(value, separator): | ||
if isinstance(value, list): | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. same There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't want to accept several different types of inputs, it should be either a value or a list. Passing an iterable, like a set, that is not ordered could produce unexpected results. Lets keep the interface as narrow as possible, there is always time to extend if we want. |
||
result = [] | ||
for v in value[:-1]: | ||
result.append(v) | ||
result.append(separator) | ||
result.extend(value[-1:]) | ||
return result | ||
else: | ||
return [value] | ||
|
||
def define(self, name, value, separator=" "): | ||
value = self._list_value(value, _Sep(separator)) | ||
self._values[name] = value | ||
|
||
def define_path(self, name, value): | ||
value = self._list_value(value, _PathSep) | ||
self._values[name] = value | ||
|
||
def unset(self, name): | ||
""" | ||
clears the variable, equivalent to a unset or set XXX= | ||
""" | ||
self._values[name] = [] | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Does it mean that the environment variable is assigned an empty value or that it doesn't exist? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The value exists in the |
||
|
||
def append(self, name, value, separator=" "): | ||
value = self._list_value(value, _Sep(separator)) | ||
self._values[name] = [_EnvVarPlaceHolder] + [_Sep(separator)] + value | ||
|
||
def append_path(self, name, value): | ||
value = self._list_value(value, _PathSep) | ||
self._values[name] = [_EnvVarPlaceHolder] + [_PathSep] + value | ||
|
||
def prepend(self, name, value, separator=" "): | ||
value = self._list_value(value, _Sep(separator)) | ||
self._values[name] = value + [_Sep(separator)] + [_EnvVarPlaceHolder] | ||
|
||
def prepend_path(self, name, value): | ||
value = self._list_value(value, _PathSep) | ||
self._values[name] = value + [_PathSep] + [_EnvVarPlaceHolder] | ||
|
||
def save_bat(self, filename, generate_deactivate=False, pathsep=os.pathsep): | ||
deactivate = textwrap.dedent("""\ | ||
echo Capturing current environment in deactivate_{filename} | ||
setlocal | ||
echo @echo off > "deactivate_{filename}" | ||
echo echo Restoring environment >> "deactivate_{filename}" | ||
for %%v in ({vars}) do ( | ||
set foundenvvar= | ||
for /f "delims== tokens=1,2" %%a in ('set') do ( | ||
if "%%a" == "%%v" ( | ||
echo set %%a=%%b>> "deactivate_{filename}" | ||
set foundenvvar=1 | ||
) | ||
) | ||
if not defined foundenvvar ( | ||
echo set %%v=>> "deactivate_{filename}" | ||
) | ||
) | ||
endlocal | ||
|
||
""").format(filename=filename, vars=" ".join(self._values.keys())) | ||
capture = textwrap.dedent("""\ | ||
@echo off | ||
{deactivate} | ||
echo Configuring environment variables | ||
""").format(deactivate=deactivate if generate_deactivate else "") | ||
result = [capture] | ||
for varname, varvalues in self._values.items(): | ||
value = self._format_value(varname, varvalues, "%{name}%", pathsep) | ||
result.append('set {}={}'.format(varname, value)) | ||
|
||
content = "\n".join(result) | ||
save(filename, content) | ||
|
||
def save_ps1(self, filename, generate_deactivate=False, pathsep=os.pathsep): | ||
# FIXME: This is broken and doesnt work | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. what exactly is broken here? also it doesn't seem to be used anywhere at all There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The whole powershell implementation is not working neither tested. I need a PS1 expert to contribute it, maybe you or @solvingj can do something, but that can wait, it is not central to this PR. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe a There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'll be happy to work on the PS1 implementation when ready. |
||
deactivate = "" | ||
capture = textwrap.dedent("""\ | ||
{deactivate} | ||
""").format(deactivate=deactivate if generate_deactivate else "") | ||
result = [capture] | ||
for varname, varvalues in self._values.items(): | ||
value = self._format_value(varname, varvalues, "$env:{name}", pathsep) | ||
result.append('$env:{}={}'.format(varname, value)) | ||
|
||
content = "\n".join(result) | ||
save(filename, content) | ||
|
||
def save_sh(self, filename, generate_deactivate=False, pathsep=os.pathsep): | ||
deactivate = textwrap.dedent("""\ | ||
echo Capturing current environment in deactivate_{filename} | ||
echo echo Restoring variables >> deactivate_{filename} | ||
for v in {vars} | ||
do | ||
value=$(printenv $v) | ||
if [ -n "$value" ] | ||
then | ||
echo export "$v=$value" >> deactivate_{filename} | ||
else | ||
echo unset $v >> deactivate_{filename} | ||
fi | ||
done | ||
echo Configuring environment variables | ||
""".format(filename=filename, vars=" ".join(self._values.keys()))) | ||
capture = textwrap.dedent("""\ | ||
{deactivate} | ||
echo Configuring environment variables | ||
""").format(deactivate=deactivate if generate_deactivate else "") | ||
result = [capture] | ||
for varname, varvalues in self._values.items(): | ||
value = self._format_value(varname, varvalues, "${name}", pathsep) | ||
if value: | ||
result.append('export {}="{}"'.format(varname, value)) | ||
else: | ||
result.append('unset {}'.format(varname)) | ||
|
||
content = "\n".join(result) | ||
save(filename, content) | ||
|
||
def compose(self, other): | ||
""" | ||
:type other: Environment | ||
""" | ||
for k, v in other._values.items(): | ||
existing = self._values.get(k) | ||
if existing is None: | ||
self._values[k] = v | ||
else: | ||
try: | ||
index = v.index(_EnvVarPlaceHolder) | ||
except ValueError: # The other doesn't have placeholder, overwrites | ||
self._values[k] = v | ||
else: | ||
new_value = v[:] # do a copy | ||
new_value[index:index + 1] = existing # replace the placeholder | ||
# Trim front and back separators | ||
val = new_value[0] | ||
if isinstance(val, _Sep) or val is _PathSep: | ||
new_value = new_value[1:] | ||
val = new_value[-1] | ||
if isinstance(val, _Sep) or val is _PathSep: | ||
new_value = new_value[:-1] | ||
self._values[k] = new_value | ||
return self | ||
|
||
|
||
class ProfileEnvironment: | ||
def __init__(self): | ||
self._environments = OrderedDict() | ||
|
||
def __repr__(self): | ||
return repr(self._environments) | ||
|
||
def get_env(self, ref): | ||
""" computes package-specific Environment | ||
it is only called when conanfile.buildenv is called | ||
""" | ||
result = Environment() | ||
for pattern, env in self._environments.items(): | ||
if pattern is None or fnmatch.fnmatch(str(ref), pattern): | ||
env = self._environments[pattern] | ||
memsharded marked this conversation as resolved.
Show resolved
Hide resolved
|
||
result = result.compose(env) | ||
return result | ||
|
||
def compose(self, other): | ||
""" | ||
:type other: ProfileEnvironment | ||
""" | ||
for pattern, environment in other._environments.items(): | ||
existing = self._environments.get(pattern) | ||
if existing is not None: | ||
self._environments[pattern] = existing.compose(environment) | ||
else: | ||
self._environments[pattern] = environment | ||
|
||
@staticmethod | ||
def loads(text): | ||
result = ProfileEnvironment() | ||
for line in text.splitlines(): | ||
line = line.strip() | ||
if not line or line.startswith("#"): | ||
continue | ||
for op, method in (("+=", "append"), ("=+", "prepend"), | ||
("=!", "unset"), ("=", "define")): | ||
tokens = line.split(op, 1) | ||
if len(tokens) != 2: | ||
continue | ||
pattern_name, value = tokens | ||
pattern_name = pattern_name.split(":", 1) | ||
if len(pattern_name) == 2: | ||
pattern, name = pattern_name | ||
else: | ||
pattern, name = None, pattern_name[0] | ||
|
||
env = Environment() | ||
if method == "unset": | ||
env.unset(name) | ||
else: | ||
if value.startswith("(path)"): | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't like this magic keyword, but I understand it is convenient to inform Conan it is a path... have we considered any other alternative? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I tried a few alternatives, like different operators, but this was the most readable and intuitive and easy to parse at the same time, I think it is difficult to beat this, but suggestions welcome:
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. My concern is that
maybe we need to rely on comments?
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The problem is that there might be env-vars that need to contain a # too, so lines cannot contain comments either. Strictly speaking, the only character that cannot be an env-var name is As the environment variable values can contain any arbitrary value, then there is by definition impossible to have any If we are concerned about |
||
value = value[6:] | ||
method = method + "_path" | ||
getattr(env, method)(name, value) | ||
|
||
existing = result._environments.get(pattern) | ||
if existing is None: | ||
result._environments[pattern] = env | ||
else: | ||
result._environments[pattern] = existing.compose(env) | ||
break | ||
else: | ||
raise ConanException("Bad env defintion: {}".format(line)) | ||
return result |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We have
conans.utils.misc.make_tuple
that will work here (pay attention, a string is iterable)There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See comment below,
filename
must be either a string value or a list. Other input types as sets, tuples, dicts, etc are not supported, and it will be documented as this.