Connection string in .env-file not read or parsed properly

[timovp]

Hi,

I have an azure blob storage connection string stored in my .env file, as follows:

CONNECT_STRING_BLOB="DefaultEndpointsProtocol=https;AccountName=xxxx;AccountKey=asdfasdfasdf/asdfasdfdsafdas/asdfadsfdsaf/ads==;EndpointSuffix=core.windows.net;"

Recently I'm experiencing very spotty behavior where me and/or colleagues on similar setups have different results:
Sometimes parsed as DefaultEndpointsProtocol sometimes as the full connect string.

• Workaround has been: to replace CONNECT_STRING_BLOB to CONNECTSTRINGBLOB, but implementing this change everywhere is not an option, and more importantly it does not always solves the issue
• Workaround has been as well to remove "_" and place it back, but that does not work always either.
• Executing the file with the debugger in VS Code solves the issue somewhat consistently.

Not sure even if this is a python-dotenv issue, but I have litterally no clue anymore where to look as it's so inconsistent and cant seem to find a solution for it.

We have the same requirements file installed, including latest python-dotenv.
Python 3.8.10
Ubuntu-20.04 on WSL1
working in Visual Studio Code (if that even matters)

[epomatti]

Same here, results testing with this .env:

# This fails and I get value "InstrumentationKey".
APPLICATION_INSIGHTS_INSTRUMENTATION_KEY='InstrumentationKey=00000000-0000-0000-0000-000000000000;IngestionEndpoint=https://eastus-8.in.applicationinsights.azure.com/;LiveEndpoint=https://eastus.livediagnostics.monitor.azure.com/'

# This works and I get the correct full value.
APPLICATION_INSIGHTS_CONNECTION_STRING='InstrumentationKey=00000000-0000-0000-0000-000000000000;IngestionEndpoint=https://eastus-8.in.applicationinsights.azure.com/;LiveEndpoint=https://eastus.livediagnostics.monitor.azure.com/'

Simple test code:

import os
from dotenv import load_dotenv
load_dotenv()

api_key = os.environ['APPLICATION_INSIGHTS_INSTRUMENTATION_KEY']
# api_key = os.environ['APPLICATION_INSIGHTS_CONNECTION_STRING']

print(api_key)

[htfra]

Same problem here as well.
Stopped working in windows 11 and python 3.11.
Then i switched to using WSL Debian, worked fine for a few days before that to stopped working.
I have colleague working on MacOS where it is working with no issues using dotenv on same project.

[Bajron]

I tried to reproduce this, but with no luck.

Could you confirm the example from https://gist.github.com/Bajron/da5ce70430c0e071bc1b68161a311d6b is enough to replicate the problem in your environments? This is mostly the example from @epomatti.

I run it like this

git clone https://gist.github.com/da5ce70430c0e071bc1b68161a311d6b.git python-dotenv-496 && cd python-dotenv-496
# assuming linux 
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
python test.py

The common denominator so far seems to be the Windows and WSL.

[htfra]

Windows 11 23H2 and python 3.11.4

WSL
Distributor ID: Debian
Description: Debian GNU/Linux 12 (bookworm)
Release: 12
Codename: bookworm

Got my colleague to clone repo and do the same test, it does not fail for him running MacOs.

[Bajron]

I am on Windows 10 22H2, tried in cmd, powershell, wsl Ubuntu, wsl Debian. Cannot get the problem.
Used python 3.11.1, now updated to 3.11.7, still nothing.

If you can get it consistently it would be great if you could debug it a little bit more.

My current guesses are:

• there is something in the system that interferes with reading the file (other app processing .env, antivirus, OneDrive?); the fact the value is cut off right before the = is very peculiar
• something interferes with the regexes used in parsing (rather unlikely, I think it is a part of Python, not delegated elsewhere)

I have added test_verbose.py to the gist. It simply checks the contents of the file. Let me know if it shows anything suspicious.

[htfra]

It happens consistently on all versions of Python from 3.9 to 3.12.
Most likely a windows issue, but i am not able to uninstall windows updates to test.

[Bajron]

Note the dotenv_values() output is ok in this case, so reading file and parsing seems fine. (perhaps you could use it as a workaround in your application for now)

It seems the problem appears in setting the variable in environment or retrieving it back.

Added test_env.py to the gist with simple os.environ actions.
It would be great if you could also test this very variable name and the value in Windows shells. For example in cmd

set APPLICATION_INSIGHTS_INSTRUMENTATION_KEY
set APPLICATION_INSIGHTS_INSTRUMENTATION_KEY='InstrumentationKey=00000000-0000-0000-0000-000000000000;IngestionEndpoint=https://eastus-8.in.applicationinsights.azure.com/;LiveEndpoint=https://eastus.livediagnostics.monitor.azure.com/'
set APPLICATION_INSIGHTS_INSTRUMENTATION_KEY
echo %APPLICATION_INSIGHTS_INSTRUMENTATION_KEY%

[htfra]

[Bajron]

I am afk for some time, could you try the same without the single quotes in cmd.

Is it the same for test_env.py?

[roberto-barrero]

Hello everyone! I have a similar situation.

I'm testing Azure Cosmos DB locally with the emulator, which provides a "secret" key required for the connection (it's the same for all local installations). The key is
C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==

I'm working on Windows 11, and the connection to the database had been working correctly, but today I started having problems with my connection. The actual message error was binascii.Error: Incorrect padding, but the cause was that the value was not being read correctly.

I found this issue and as @Bajron mentioned, it seems like the values are being read correctly, but when accessing them from the variable, in my case, the padding '==' at the end of the key is removed from the value.

I tried this and got this results:

import dotenv
import os

dotenv.load_dotenv()
print("Loaded value to the environent variable:", os.getenv('COSMOS_KEY'))
print("Value obtained with dotenv_values:", dotenv.dotenv_values().get('COSMOS_KEY'))

From VS Code using the virtual environment (as seen on the bottom-right corner):

From VS Code using the global Python installation (as seen on the bottom-right corner):

Then I ran the test for this issue (https://gist.github.com/Bajron/da5ce70430c0e071bc1b68161a311d6b) directly from PowerShell and got this:

I expected it to fail, but apparently it passed the test.

Then I tried this: from VS Code, from the global Python installation, I loaded my project's venv and this was the result:

In conclusion, I'm confused.

Python 3.9.13
Windows 11 22H2
Working in Visual Studio Code
Issue tests ran with python-dotenv=1.0.1
Own tests ran with python-dotenv=1.0.0

[bill-ash]

Same issue:
wsl2
Python 3.10.12
python-dotenv=1.0.1

Load creds with load_dotenv() and strips trailing == in azure blob storage key causing the auth to fail.

When I rename the key from AZURE_KEY to AZ_KEY or anything else, it will then parse correctly for a few hours before failing again.

Tried every combination of quotes around the value, but it seems only changing the key will work, albeit temporarily. Issue began for me this last week.

Haven't been able to reproduce. Seems to quit parsing correctly at random.

tldr;
Will randomly fail to parse trailing ==
AZURE_KEY=keyvaluesasdasd==

If I change the name of the key, it will work for a few hours before failing again.
AZ_KEY=keyvaluesasdasd==

edit:

When switching from os to dotenv_values, the keys parse correctly. Super weird.

import os 
from dotenv import dotenv_values

os.environ['AZ_KEY'] # this fails to parse trailing ==
dotenv_values()['AZ_KEY'] # this parses correctly 

[Bajron]

@bill-ash could check setting the value directly to os.environ and reading it back? I mean the one that stopped working for you after some time.

So far it seems like a faulty handling of environment variables in Python or Windows (parsing in dotenv seems fine).

[htfra]

Now python-dotenv suddenly started working again for me after getting the Cumulative Update for Windows 11.

[hoisunng]

I was also having this problem and in my search I stumbled upon this issue in VS Code that seems to cause the problem for me.
microsoft/vscode-python#23078

[bbc2]

That's a good point. If VS Code is setting environment variables before the Python program is run, those variables will override what you load with load_dotenv (unless you use override=True), so even if Python-dotenv parses the .env correctly, it doesn't matter because the value from VS Code will be used.

Of course this is different with dotenv_values which ignores environment variables and only looks at the file.

Then it seems this issue might be unrelated to Python-dotenv?

[DanielRosenwasser]

With microsoft/vscode#207889, this should now be fixed in VS Code.

Would people here be able to try out Visual Studio Code Insiders possibly with a prelease version of the Python extension?

[hoisunng]

With the prerelease version of VS code the connection strings are loaded properly again for me.