-
Notifications
You must be signed in to change notification settings - Fork 954
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Spotipy doesnt doesnt respond but no error #913
Comments
I believe something occurs that spams the Spotify API and they essentially blacklist your app. I wrote my own requests function to query the API using my existing credentials I used with spotipy and it was giving me a 429 error for "Too Many Requests" even when waiting a long time between search requests. I made a new Spotify app on the website (new client/secret) and used my custom requests function and it worked perfectly. Made another Spotify app on the website (new client/secret) and tried plugging those into the existing spotipy application and was still getting an infinite hanging problem. So I think what's happening is spotipy is getting a 429 error and not handling/reporting it. It must be spamming requests at some point causing the issue, notice how you have a bump in API activity for a brief moment, my guess is spotipy unintentionally spammed the API with thousands of requests in a second, and Spotify auto-blocked your application from sending more. |
Hi! I had a similar issue. I found that once a request gets denied, with each additional call you make following this, the wait time grows exponentially. Since I had a lot of calls to make (I'm doing a dataset augmentation task), I added an error catch for time out that looks like this:
Normally, it makes me wait like 8 hours and then I can come back. Also, a few tips I have for making it less likely to time out:
Not sure whether you setting allows for these changes, but in the case that it does - hope it helps :) |
spotipy should already be waiting/sleeping when the Spotify API response requests to wait (retry-after value). Maybe there is a scenario where this does not apply. Would anyone have a Minimal reproducible example that would allow us to reproduce this rate limiting? This would need to work with a new Spotify app. Feel free to include logs about the current retry-after value, maybe with inspiration from #766 (comment)? |
Hey, I was just curious if Rsalgnakit1223's solution with the try catch blocks would work even if my spotipy api doesn't throw an error. Mine just gets stuck in sleep_for_rety() and hangs forever. Would the try catch block work in my case if an error is never thrown to trigger the catch? |
@CalColistra, sorry Rsalgnakit1223's solution is incomplete, here is a better example which will raise an exception #766 (comment) |
@stephanebruckert Hey, I am sorry if I missed anything obvious in any of the issues made for this sleep_for_retry() problem with the spotipy api. But are there any working solutions that we know of? I see you are keeping up with a lot of the discussions about this and really appreciate you taking the time to do so. |
|
It gives me a 22 hour timeout the first time of bumping into 429 in this fresh run after 23k search requests, does the API assign rate limit based on IP too? I read on previous issues that the timeout would go up exponentially every time you hit the limit. How does the timeout get determined for search under Web API? [main][INFO ]processing the 23210th title
I thought I could catch the retry_after and make the program sleep for the value, but 22 hours timeout renders that not very useful...
|
@reflexiveprophecy can you please provide a Minimal reproducible example? Something that we could just run once to reproduce the problem with a new Spotify app? The code above doesn't look complete as it doesn't include any loop.
I agree but the app is supposed to start sleeping from the very first 429s, not after 23k requests and 22 hours wait. Could your code show that as well? My assumption is that at one point, it's not "respecting" the given I imagine examples of such violation could be occurring when:
|
@stephanebruckert Okay, please see the following self-contained example. You shouldn't have to do much else to run this example besides pip install requests and embedding CLIENT_ID and CLIENT_SECRET as env variables. This example would produce 100k artist names in a list for the search request to run through in a loop. I tested the script today with a new set of credentials and was able to run through 24.9k search requests this time without a 429 error and it gives me a 23 hour timeout this time after bumping into the error this time...I don't see other 429 messages as I searched through the entire log. And I don't think it's particularly blocking my ip address either as I was able to run through again by switching to new "non-timeout" account credentials. Also I didn't use Spotipy client as the client silences 429 messages for some reasons as previous issues also noted. Looking forward to some solutions :)
|
Thanks for posting this response, it was very helpful! I'm more of a beginner, but I'm trying to collect some tracks metadata. I'm getting a 429 response, which I expected (I'm making many requests to the API), but the header comes back empty. So when I use |
@spotifyr1 Hey, regarding this question "So when I use if int(e.headers['retry-after']) > 60 to exit the loop, it doesn't work because there is no 'retry-after' key in the headers. Do you know why that could happen? I'm printing out the values returned by the exception, and they all look fine except the header which is empty {}", I don't think spotipy has this returned. This is one of the reasons why I didn't use spotipy client as the api, I directly called the REST API with the requests module, hence you don't see me importing spotipy and just the requests module. With the requests module, you should be able to catch the retry-after value. |
@reflexiveprophecy I took your advice, but I have the same issue with the requests module directly (The get command blocks forever even with a timeout applied) So I am unsure as to whether this is a problem with Spotipy. |
Hey @Drag-3, got it, happy to test it out if you could share your code, thank you! |
@reflexiveprophecy Here's my code! Hopefully you can find something I am missing. def __init__(self, cid: str, secret: str):
try:
self.auth_path = CONFIG_DIR / ".sp_auth_cache"
self.session = requests.Session()
retries = Retry(total=5, backoff_factor=0.1, status_forcelist=[429, 500, 502, 503, 504])
adapter = HTTPAdapter(max_retries=retries)
self.session.mount('https://', adapter)
self.session.mount('http://', adapter)
AUTH_URL = "https://accounts.spotify.com/api/token"
auth_resp = requests.post(AUTH_URL, {"grant_type": "client_credentials",
"client_id": cid,
"client_secret": secret})
auth_resp_json = auth_resp.json()
access_token = auth_resp_json['access_token']
self.session.headers.update({'Authorization': f'Bearer {access_token}'})
self.cache = diskcache.Cache(directory=str(CONFIG_DIR / "spotifycache"))
self.cache.expire(60 * 60 * 12) # Set the cache to expire in 12 hours
self.semaphores = {
'search': threading.Semaphore(3),
'track': threading.Semaphore(3),
'audio-analysis': threading.Semaphore(2),
'artists': threading.Semaphore(1)
}
except Exception as e:
logging.exception(e)
raise e
def _get_item_base(self, endpoint: str, value):
with self.semaphores[endpoint]:
response = self.session.get(f"https://api.spotify.com/v1/{endpoint}/{value}", timeout=20) # This line blocks according to the debugger.
if response.status_code == 429: # Added these trying to debug before I noticed the blocking problem
retry_after = int(response.headers.get('retry-after', '1'))
logging.warning(f" {endpoint} Rate limited. Waiting for {retry_after} seconds before retrying.")
time.sleep(retry_after + random.randint(3, 7))
elif response.status_code != 200:
response.raise_for_status()
return response.json() According to curl, I should be getting an Error 429, but the code blocks instead. |
I run into the same problem today with this reduced example with spotipy 2.23.0:
My mistake was: I was in headless mode, and the "open the browser" part obviously can not work there. In the example section of spotify I found that you need to set
With this setup, the oauth flow will be handled in terminal. Hope this is helpful for some 👍 |
I found the cause of the lack of response, and this is mentioned in urllib3/urllib3#2911. In module urllib3/connectionpool.py, line #943 is code retries.sleep(response) This is honoring the Retry-After that Spotify's API is sending back. And if you're like me, who somehow got a long retry time (mine is currently 20k+ seconds), it is going to hang. A potential fix is simply doing the following... sp = spotipy.Spotify(
retries=0,
...
) ..so Spotify doesn't try the retry. But if you do this, it just raises an Exception and doesn't report back what the Retry-After value was. This is where the improvement can be made, and perhaps build Spotipy's own retry feature. |
If you create your own import spotipy
from spotipy.oauth2 import SpotifyClientCredentials
import time
import requests
session = requests.Session()
sp = spotipy.Spotify(auth_manager=SpotifyClientCredentials(client_id="CLIENT_ID",
client_secret="CLIENT_SECRET"),
retries=0,
requests_session=session)
def make_request():
try:
print(sp.playlist("PLAYLIST_ID", fields="name"), time.time())
except spotipy.exceptions.SpotifyException as e:
print(e.headers["retry-after"])
while True:
make_request() Sometimes a timeout occurs, so catching (and ignoring) timeout errors would probably be necessary to run this.
I'm honestly not sure how much we can improve this. Perhaps a warning to inform the user of the current situation, as there are a lot of issues being opened stating that spotipy has just started to hang randomly. Edit: My application is still rate limited, but I've already made one observation. Every hour or two, a single request is successfully executed, and then the countdown continues as if nothing had happened. |
There seems to be some misunderstanding of this error message and very large 'retry-after' value, I get that error while fetching sp.album_tracks, regardless of the time between requests. It is not what people are understanding to be a rate limit where you just have to slow down. I think it means that there is a limit to the number of requests you can make of a specific type in any 24 hour period. You don't get any prior warning, it is a brick wall limit and lasts for multiple hours. Testing over the last few days suggests that the limit is somewhere in excess of 2000 sp.album_tracks queries in a (presumably) 24 hour period. I'll do more tests and see if the quota reset time is the same time each day, or is dependent on the time of your first / last query. There may be similar errors for other query types, but I only realised this week that this was happening as I was getting a fairly consistent set of results each day. I can't see anything on the Spotify documentation or forum that confirms my suspicion though.
I'm seeing this too. |
Hello,
I have been using spotipy for the last year and never ran into any problems.
Today, I realized that something is not working anymore and looked into it.
I can initializing the
sp.Spotify
instance as usual without any problems, but if I then call a function (for examplespotify.me()
orspotify.devices()
, it simply hangs and doesnt return anything.This is the code I used for the last months:
I checked my Spotify dashboard, and noticed, that noticed that the number of daily requests dropped to zero just when December started:
Do you have any idea what might cause the issue? I already tried to regenerate a new client secret, but it didnt work.
I am using the version 2.21.0 on a raspberry pi (Raspbian version 10 [Buster])
The text was updated successfully, but these errors were encountered: