README dev instructions are currently broken

[bjrne]

When following the developer instructions in the readme, specifically pip install -r requirements-dev.txt, the following error shows:

ERROR: File "setup.py" not found. Directory cannot be installed in editable mode: /[...]/recipe-scrapers
(A "pyproject.toml" file was found, but editable mode currently requires a setup.py based build.)

From what I understand, this comes from deleting the setup.y after the switch to the toml file in #655.

I'd update the readme with a current command, but it seems that the requirements-dev.txt file is completely outdated? It's missing at least beautifulsoup4 which is given in the toml file, among other differences. After installing all dependencies I could find, everything worked. However, after reading through #654, I'm still unsure whether there is a recommended way to install all dependencies currently? This comment also talks about dependencies, but I'm not any wiser now.

Maybe someone can suggest what they currently use for dependency management as a developer? I'll try to update the readme.

[hhursev]

Hey, thanks for taking the time to notify us!

Can you provide us with more information?

• what OS you are on
• python and pip versions (after you activate the venv)

I suspect your pip version is dated. Will go out on a limb if I were you and upgrade it python -m pip install --upgrade pip. I think you won't be facing problems doing so 🤞 .

Please let us know if pip being older version was indeed the problem so we can update the README accordingly.

[bjrne]

Thanks! You were correct indeed. I am using ubuntu 20.04 LTS with python3.8 and originally pip 20.0.2, which was upgraded to 22.3.1 (newest) and works now. Weirdly enough, I don't remember pip ever notifying me of a new version. Feel free to close this, thanks for your time and effort!

I am thinking about supporting this library a bit more, is there some coordination going on between the contributors somewhere? I see some open issues with varying degrees of completeness. There is technical debt as well as discussions about (testing) the quality of scraper outputs (#162). Is there some ranked list with priorities or people to talk to?

[hhursev]

Glad to hear you are willing to contribute to the lib!

I don't think we have a clear coordination who does what or clear priority of the tasks/issues one can per take.

@jayaddison may have a better idea what would be a good first issue(s) for you tackle on.

I'll think over this weekend how we can coordinate better going forwards.

[jayaddison]

Hi @bjrne! Welcome. We don't really have any chat/forum co-ordination for this library, nor task priorities - communication is through GitHub issues and pull requests, and although I can't speak for other contributors, I mostly work on things when I feel there's some need to do that (emergent priority, rather than planned, I suppose).

I probably broke the README while trying to make it easier for new developers to onboard, argh - see #617 for a long and ongoing conversation about some of the options and what has happened so far.

[jayaddison]

@bjrne do you have any experience with managing software projects that have multiple version branches, out of interest?

(the reason I ask: I started a v15 branch a little while ago, but am slightly worried it may go out-of-sync or become difficult to keep up-to-date with main)

Any time taking a look at that and providing feedback would be useful (for me at least; hopefully for the library too!)

[jayaddison]

And to explain the reasoning behind having a v15 branch in the first place: as mentioned in #617, we'd like to remove the requests dependency from the library at some point - but doing that will break the existing scrape_me interface, so it's probably best to put that in a major version update.

[jayaddison]

@bjrne actually, one more (hopefully easier and perhaps more rewarding) idea: running pylint on this codebase and finding improvements to apply could be worthwhile.

My guess is that it might report a lot of things to begin with - that's ok, we can start gradually and improve from there.

If that proves worthwhile we could add it to the list of items in #617.

[jayaddison]

(or any other Python linter you prefer)

[bjrne]

Thanks for keeping me up to date, I have been sick for some time now and will get back to you soon :) But I do have experience with git and parallel feature development.

[jayaddison]

There's no hurry, thank you; hope you feel better soon.