Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add V2 sources and feed files #672

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from
Draft

Conversation

aurangzaib048
Copy link
Member

@aurangzaib048 aurangzaib048 commented Nov 30, 2023

Related to #678

This will add new files feed.{LOCALE}.v2.json and sources.global.v2.json which excludes the un-used fields as well as format the feed according to the new format.

  • keys_to_remove_v2 are the keys which will be excluded from the feed.locale.v2.json.
  • publisher_include_keys_v2 are the keys which will be the parts of sources.global.v2.json.

New v2 format of the feed will be like:

{
    "3e4b01d03972894295e34f8dbc395745395641ffb3c58d7ec4aab0c545cfac1d":
    [
        {
            "title": "Christmas Toy Charity in Western Michigan Turns to Gift Cards After Fire",
            "publish_time": "2023-11-29 20:09:20",
            "description": "A charity that has been giving away Christmas gifts in the Grand Rapids area for more than 100 years said it will turn to gift cards next week",
            "content_type": "article",
            "publisher_id": "3e4b01d03972894295e34f8dbc395745395641ffb3c58d7ec4aab0c545cfac1d",
            "channels":
            [
                "US News"
            ],
            "url": "https://www.usnews.com/news/best-states/michigan/articles/2023-11-29/christmas-toy-charity-in-western-michigan-turns-to-gift-cards-after-fire",
            "pop_score": 1,
            "padded_img": "",
            "score": 15.403737755255854
        },
        {
            "title": "Kraft Debuts Dairy-Free Mac and Cheese in the US",
            "publish_time": "2023-11-29 20:08:38",
            "description": "Kraft Heinz says it’s bringing dairy-free macaroni and cheese to the U.S. for the first time",
            "content_type": "article",
            "publisher_id": "3e4b01d03972894295e34f8dbc395745395641ffb3c58d7ec4aab0c545cfac1d",
            "channels":
            [
                "US News"
            ],
            "url": "https://www.usnews.com/news/best-states/illinois/articles/2023-11-29/kraft-debuts-dairy-free-mac-and-cheese-in-the-us",
            "pop_score": 1,
            "padded_img": "",
            "score": 30.882698512023513
        }
    ],
    "9dfe110b0027d7bcbab5be6f42f4512172747474b293449ea25eb5239885c7a2":
    [
        {
            "title": "U.A.W. Announces Drive to Organize Nonunion Plants",
            "publish_time": "2023-11-29 20:10:07",
            "description": "The United Automobile Workers’ effort, with a long-elusive goal, follows its success in securing big raises in contracts with the Detroit automakers.",
            "content_type": "article",
            "publisher_id": "9dfe110b0027d7bcbab5be6f42f4512172747474b293449ea25eb5239885c7a2",
            "channels":
            [
                "US News",
                "Top Sources",
                "Top News"
            ],
            "url": "https://www.nytimes.com/2023/11/29/business/uaw-union-organizing.html",
            "pop_score": 1,
            "padded_img": "https://static01.nyt.com/images/2023/11/29/multimedia/29uaw-ktmw/29uaw-ktmw-mediumSquareAt3X.jpg",
            "score": 15.360765678585501
        },
        {
            "title": "Iranian and U.S. Attacks Could Lead to Larger War, Officials Say",
            "publish_time": "2023-11-29 19:04:19",
            "description": "U.S. forces and militias backed by Iran have launched tit-for-tat attacks, fueling concerns as Israeli forces have also clashed with Iranian-backed groups.",
            "content_type": "article",
            "publisher_id": "9dfe110b0027d7bcbab5be6f42f4512172747474b293449ea25eb5239885c7a2",
            "channels":
            [
                "US News",
                "Top Sources",
                "Top News"
            ],
            "url": "https://www.nytimes.com/2023/11/29/us/politics/israel-iran-gaza-us-attacks.html",
            "pop_score": 1,
            "padded_img": "https://static01.nyt.com/images/2023/11/28/multimedia/00dc-military-01-tvgl/00dc-military-01-tvgl-mediumSquareAt3X.jpg",
            "score": 34.873014623488885
        }
    ]
}

Fix:

Copy link

The following commits were not verified:
c2e6b90 (unsigned)

keys_to_remove_v2 = [
"category",
"url_hash",
"img",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it would be good to keep this field instead of padded_img as its name it's a bit misleading. The great majority of times padded_img is a url to a non-padded image anyways

@brave brave deleted a comment from Noodlezzzzzz Dec 29, 2023
@aurangzaib048 aurangzaib048 marked this pull request as draft February 12, 2024 16:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants