As a user, I can query a single keyword and get its Google search raw response

[malparty]

Why?

In order for the user to manually analyse results (AdWords, links, etc...).

Additional notes:

Problems to solve:

• AdWords link are included in the result (google remove adwords for bot requests!!)
• Each query executes in less than 1s
• Results come from https://google.com/search

No Gos

• No data persistence [yet]
• No query parsing [yet]
• No CSV File upload/parsing [yet]
• No fancy UI as such UI would not be sustained

Who Benefits?

Logged users.

[junan]

@malparty Uploading the csv file will be done here? or some other ticket?

[malparty]

@junan , it will be done later, in #9 [BackEnd] & #13 [FrontEnd].
The main idea is that I can focus first on the Google query/parsing (which contains more unknows than csv file upload/parsing). But it will be deployable and fully testable. ;-)
The UI though will be minimal (that's why no FrontEnd task for this one) because it will be replaced later by the file upload.

[junan]

@malparty Another question, will you handle this requirement in this ticket or in some other ticket?

We mostly process google search request/response/parsing stuff in a background job and add delay between each google search request to avoid google's IP blocking for massing requests

[ankitkalia1195]

@malparty IMO this task is kind of unneeded in the IC requirements, because we ulimately require parsing of the results and showing those results. I understand this idea.

The main idea is that I can focus first on the Google query/parsing (which contains more unknows than csv file upload/parsing). But it will be deployable and fully testable. ;-)

But then if you want to focus google query/parsing first, why create the UI? Just do the backend part, or for this case just creating a google client class which will make request to google and get raw response. No need for UI. You can test your code by running form console and writing tests for it, do you need to build a UI just to test that part 🤔 In our client projects too, we dont build a UI which will not be used the user, just for developer convinience.
What do you think 🤔

    class GoogleClient
        def get(keyword)
              
        end
    end

[malparty]

her question, will you handle this requirement in this ticket or in some other ticket?

We mostly process google search request/response/parsing stuff in a background job and add delay between each google search request to avoid google's IP

@junan
This will be done later when by the background worker(once CSV files will be uploadable).
In my previous C# project, it was impossible for me to get "Ads" in my queries, that's the purpose of this issue: perform a single query that returns results with ads.

[malparty]

@malparty IMO this task is kind of unneeded in the IC requirements, because we ulimately require parsing of the results and showing those results. I understand this idea.

The main idea is that I can focus first on the Google query/parsing (which contains more unknows than csv file upload/parsing). But it will be deployable and fully testable. ;-)

But then if you want to focus google query/parsing first, why create the UI? Just do the backend part, or for this case just creating a google client class which will make request to google and get raw response. No need for UI. You can test your code by running form console and writing tests for it, do you need to build a UI just to test that part 🤔 In our client projects too, we dont build a UI which will not be used the user, just for developer convinience.
What do you think 🤔

    class GoogleClient
        def get(keyword)
              
        end
    end

@ankitkalia1195
True, I created the backlog before getting to know Rails well. Back in the C# days, we did not have the "Rails console" flexibility and creating minimal entry points enabled to demo the result (aka getting feedbacks from the client == which is the purpose of a User Story, no? :) ).

Here with Rails Console, we can indeed perform some queries and show the results to our client, so I agree it works perfectly without UI.

[junan]

@junan
This will be done later when by the background worker(once CSV files will be uploadable).
In my previous C# project, it was impossible for me to get "Ads" in my queries, that's the purpose of this issue: perform a single query that returns results with ads.

@malparty You can create a separate ticket for it if you have not created already 👍

[malparty]

@junan
This will be done later when by the background worker(once CSV files will be uploadable).
In my previous C# project, it was impossible for me to get "Ads" in my queries, that's the purpose of this issue: perform a single query that returns results with ads.

@malparty You can create a separate ticket for it if you have not created already 👍

@junan
According to my initial plan, this was part of #9 - I believe it's hard to deliver this task without a background worker.
Or do you mean I should remove the "To run multiple keyword searches" part from #9 (thus having ony CSV File upload) and having another task for the "run multiple keyword searches" ?

Thanks,

[junan]

According to my initial plan, this was part of #9 - I believe it's hard to deliver this task without a background worker.
Or do you mean I should remove the "To run multiple keyword searches" part from #9 (thus having ony CSV File upload) and having another task for the "run multiple keyword searches" ?

@malparty We usually create tickets and work like this ...

1. For the CSV uploading task(its [Backend] As a User, I can upload a csv file in order to run multiple keyword searches  #9 for your case) we usually just upload the CSV file, then parse it, then store all keywords in the database.

2. Then we create another task to handle keyword searching google(this stage we already have keywords in the database, which was done in number 1), parse the response, and store it in a database.

3. Then create a separate ticket to integrate a cron job to process the number 2 implementation(Then we create ...) like [Backend] Work around the limitations of mass-searching keywords as Google prevents it carryall/go-google-scraper-challenge#67, [backend] Process csv keyword searching in the background junan/google-scraper#14

You don't need to follow exactly like this, you can plan backlog your own. By convention we just try to breakdown the ticket as much as possible and keep PR changes smaller

[malparty]

According to my initial plan, this was part of #9 - I believe it's hard to deliver this task without a background worker.
Or do you mean I should remove the "To run multiple keyword searches" part from #9 (thus having ony CSV File upload) and having another task for the "run multiple keyword searches" ?

@malparty We usually create tickets and work like this ...

1. For the CSV uploading task(its [Backend] As a User, I can upload a csv file in order to run multiple keyword searches  #9 for your case) we usually just upload the CSV file, then parse it, then store all keywords in the database.
2. Then we create another task to handle keyword searching google(this stage we already have keywords in the database, which was done in number 1), parse the response, and store it in a database.
3. Then create a separate ticket to integrate a cron job to process the number 2 implementation(Then we create ...) like carryall/go-google-scraper-challenge#67, junan/google-scraper#14

You don't need to follow exactly like this, you can plan backlog your own. By convention we just try to breakdown the ticket as much as possible and keep PR changes smaller

Well noted!
Yes, I need to change the abstraction level of "tasks" to match the criteria "easy PR". It somehow feels weird as per my definition of a User Story, but I believe here it's more "a set of several tasks that together build a user story" (same reason why there are 2 tasks for Backend/Frontend for instance).

Thanks for the clarification!
So I've created #38 to reflect this.