Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support retry-after in FetcherBolt #784

Open
jnioche opened this issue Jan 10, 2020 · 1 comment
Open

support retry-after in FetcherBolt #784

jnioche opened this issue Jan 10, 2020 · 1 comment

Comments

@jnioche
Copy link
Contributor

jnioche commented Jan 10, 2020

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Retry-After

if a site is asking for a delay, we should enforce it in the internal queues of the FetcherBolt.

@jnioche jnioche added this to the 1.17 milestone Feb 14, 2020
@jnioche
Copy link
Contributor Author

jnioche commented Jul 15, 2020

Could use the snippet below

// has a retry after? 503 (Service Unavailable) or 429 (Too Many Requests) 
                	// but also a permanent redir (301)
                    String retry_after = mergedMD.getFirstValue("Retry-After");
                    if (retry_after != null){
                    	// in seconds?
                    	if (retry_after.matches("[0-9]+")) {
                    		int rety_after_secs = Integer.parseInt(retry_after);
                    	}
                    	else {
                    		// expressed as a date
                    		// or a delay in secs
                    		SimpleDateFormat format = new SimpleDateFormat("EEE, dd MMM yyyy HH:mm:ss zzz");
                    		try {
                    			Date d = format.parse(retry_after);
                    			// pass the date
                    			d.toInstant().toString();
                    		}
                    		catch (ParseException pe) {
                    			LOG.debug("Incorrect date found for retry-after: {} - {}",retry_after, url);
                    		}
                    	}
                    }

@jnioche jnioche removed this from the 1.17 milestone Jul 17, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant