Investigate simplifying static checks for initialized Crawler

[wRAR]

After we changed the Crawler logic to initialize components such as Crawler.stats later, static checkers (specifically mypy) don't know that when code like self.crawler.stats.inc_value() is called, self.crawler.stats is already initialized, so we need to add asserts in all methods that do this. We should try to find a way to do the same with fewer asserts. This also applies to 3rd-party code (middlewares etc.).

[antarahebbar]

Hi! @yueric985 and I are students are we are working on a project for a class that requires us to make an open-source contribution to a library. We will be working on this issue as an example.

[sa2415]

Hi! Can I work on this?
"We should try to find a way to do the same with fewer asserts." --> Does this mean we'd need to reduce the assert checks in certain places or keep the assertion checks as is and simplify the way these checks are done?

[wRAR]

Hi! Can I work on this?

Sure.

Does this mean we'd need to reduce the assert checks in certain places or keep the assertion checks as is and simplify the way these checks are done?

This means doing fewer asserts.

[derekk10]

Question about the issue. I understand that several components of Crawler are initialized later. However, is it ensured that these components such as self.crawler.stats is always initialized before calling a method such as self.crawler.stats.inc_value()? If so, would doing something like self.stats = cast(StatsCollector, self.stats) be a valid solution? This would tell mypy to treat self.stats as an ainstance of StatsCollector for the rest of the method so we wouldn't have to use an assert statement before accessing self.stats.

[wRAR]

would doing something like self.stats = cast(StatsCollector, self.stats) be a valid solution?

I don't think this code does anything?

Also, where would you put it?

This would tell mypy to treat self.stats as an ainstance of StatsCollector for the rest of the method

Oh, you mean in each method? That's the same amount of code as an assert.

[derekk10]

Yes, I see the issue. I have another proposed solution to use a proxy object for crawler.stats that initializes the actual stats object the first time it is accessed. This way, we would not need to use assrts or if statements to check if stats is initialized.
It might look something like this:

`class StatsProxy:
def init(self, crawler):
self._crawler = crawler
self._stats = None

def __getattr__(self, name):
    if self._stats is None:
        self._stats = self._crawler.initialize_stats()
    return getattr(self._stats, name)

class Crawler:
def init(self):
self.stats = StatsProxy(self)

def initialize_stats(self):
    #init logic
    return Stats()`

StatsProxy intercepts all attribute accesses to crawler.stats and initializes stats the first time it's accessed. This way we can be sure that stats is always initialized before use without using asserts. I have a sample initialize_stats() method that I haven't filled out. Please let me know if this make sense and I'd appreciate feedback and your thoughts on this solution.

[wRAR]

initializes the actual stats object the first time it is accessed

That's the problem, it shouldn't be initialized on access, it's initialized by a separate method and the question of this ticket is how can we tell the checkers that.