update(parser) - ELEXON #6441

ghost · 2024-02-02T13:58:16Z

Issue

Description

Here is a diagram of how the data is collected

We assume that the hydro storage from ESO is only pumping (from the grid to the reservoir) because it happens in the night and B1620 is discharge (from the reservoir to the grid) because it happens during the day at what see to be peak hours

This means that storage from B1620 is counted as negative and storage from ESO is counted as >0

Preview

Double check

I have tested my parser changes locally with poetry run test_parser "zone_key"
I have run pnpx prettier@2 --write . and poetry run format in the top level directory to format my changes.

VIKTORVAV99 · 2024-02-07T09:30:53Z

Have you seen this new Elexon API?

5 min production: https://developer.data.elexon.co.uk/api-details#api=prod-insol-insights-api&operation=get-datasets-fuelinst

Day ahead demand: https://developer.data.elexon.co.uk/api-details#api=prod-insol-insights-api&operation=get-datasets-inddem-stream

Day ahead generation: https://developer.data.elexon.co.uk/api-details#api=prod-insol-insights-api&operation=get-datasets-indgen-stream

I have not looked through them all yet but there could be some nice gems in there.

ghost · 2024-02-07T09:33:55Z

Have you seen this new Elexon API? developer.data.elexon.co.uk/api-details#api=prod-insol-insights-api&operation=get-datasets-fuelinst

I have not looked through them all yet but there could be some nice gems in there.

yes actually the revamp of the parser uses the new API. It's just that i still need to look at the pumping data before the PR is ready for review

VIKTORVAV99

Lots of small notes that needs to be addressed.

I'm also seeing a lot of these in the terminal:

WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`
WARNING:test_parser:GB reported total of 0.00MW falls outside range of (100, 60000) for key `gas`

Is this expected? I don't see why they would report 0 gas (and nuclear).

parsers/ELEXON.py

VIKTORVAV99 · 2024-02-20T15:15:35Z

parsers/ELEXON.py

+    storage_breakdown = ProductionBreakdownList(logger=logger)
+    for event in eso_data:
+        event_datetime = parse_datetime(
+            event.get("SETTLEMENT_DATE"), event.get("SETTLEMENT_PERIOD")


While it never should be none we need to make handle it to satisify the typing (with isinstance for example).

parsers/ELEXON.py

ghost · 2024-02-21T08:57:57Z

will remove this parser validation. I don't think it makes a lot of sese

VIKTORVAV99 · 2024-02-21T13:27:46Z

I'll look into this further but there seems to be something seriously wrong with the earlier data:

{'correctedModes': [],
  'datetime': datetime.datetime(2024, 2, 19, 0, 0, tzinfo=datetime.timezone.utc),
  'production': {'biomass': 0.0,
                 'coal': 0.0,
                 'gas': 0.0,
                 'hydro': 0.0,
                 'nuclear': 0.0,
                 'oil': 0.0,
                 'solar': 0.0,
                 'unknown': 0.0,
                 'wind': 10108.67},
  'source': 'elexon.co.uk, nationalgrideso.com',
  'sourceType': <EventSourceType.measured: 'measured'>,
  'storage': {'hydro': 11.0},
  'zoneKey': 'GB'}

(This is after I changed so 0 is not considered false so we get 0 solar at night).
I think this stems from the fact that we use two different endpoints? And therefore there is a misalignment?

VIKTORVAV99

If I go to https://data.elexon.co.uk/bmrs/api/v1/datasets/FUELHH/stream?publishDateTimeFrom=2024-02-10&publishDateTimeTo=2024-02-11 everything seems to be fine data wise.

But if I run poetry run test_parser GB --target_datetime="2024-02-11" I get a bunch of zero filled data which indicated there is something flawed in either the datasets or our logic.

Turns out it's probably both. I have adressed them in individual comments.

parsers/ELEXON.py

VIKTORVAV99 · 2024-02-21T14:16:11Z

parsers/ELEXON.py

+        data = ProductionBreakdownList.merge_production_breakdowns(
+            [data, parsed_hydro_storage_data], logger, matching_timestamps_only=True
+        ).to_list()


Here we are once again merging these datapoints that was merged in the query_and_merge_production_fuelhh_and_eso function, effectivly merging them twice.

no that is not what is happening here. We are only collecting hydro storage data from the eso source and merging it with the B1620 data that only has discharge so it is not double counting

parsers/ELEXON.py

q-- · 2024-02-21T23:06:21Z

Adding a link to #6419 so its relationship to this pull request is clear in GitHub (though I'm unclear on whether this pull request addresses the issue, or that it was merely looked into).

ghost · 2024-02-22T10:04:59Z

converting back to draft, i will address this next week.

ghost · 2024-03-27T14:28:43Z

Adding a link to #6419 so its relationship to this pull request is clear in GitHub (though I'm unclear on whether this pull request addresses the issue, or that it was merely looked into).

thanks for the link. This is indeed an answer to the issue you raised about discharging at night :)

From my investigation, it seems that ESO data only covers pumping and not discharge and that is why the data is off, it is missing the other half of the puzzle!

ghost · 2024-03-27T14:58:46Z

VIKTORVAV99

Seems to be working but there is still the issue of no data at night (it's missing when it should be 0).

parsers/ELEXON.py

VIKTORVAV99 · 2024-03-27T17:14:37Z

parsers/ELEXON.py

-                report["expected_fields"], len(df.columns)
+        exchange_data = query_elexon(
+            ELEXON_URLS["exchange"], session, exchange_params
+        ).get("data")


There is no guarantee that get is available since the response is a JSON response, we need to ensure it is a dict before calling get on it.

The type data also says it will return a list, which causes type issues.

This is just a type issue so ignoring it for now since we need to get this merged ASAP.

…rser

VIKTORVAV99

I did some small changes and now the output looks a bit better with less errors by correcting negative values with 0 (so they are still valid) and by only merging the matching timestamps (so we ensure we get complete data before the validation method kicks in).

There might still be an issue with no solar at all during the night (not null or 0, simply missing) but it's not blocking from my side. So I'll leave that up to you.

I'm going to bussy for the rest of this and next week so I won't be able to fix that particular issue if you consider it blocking.

VIKTORVAV99 · 2024-05-29T12:48:16Z

@tonypls @madsnedergaard could you just give this a quick double check?
I know there is a lot of changes and they should all be fine but it would be nice to get another set of eyes on it. We need to merge this before the 1st so it's kind of urgent as they will be shutting down the old API.

So I'll be merging this tomorrow no matter what so we can monitor it during Friday if needed.

madsnedergaard

LGTM, thanks both of you for the work here ❤️

Mathilde Daugy added 3 commits January 31, 2024 16:56

elexon parser

7e1412e

elexon revamp

edf44c8

other elexon update

1f37e65

github-actions bot added parser python Pull requests that update Python code labels Feb 2, 2024

ghost marked this pull request as ready for review February 20, 2024 14:41

ghost requested a review from VIKTORVAV99 as a code owner February 20, 2024 14:41

Merge branch 'master' into md/update-elexon-parser

859636a

ghost changed the title ~~elexon parser~~ update(parser) - ELEXON Feb 20, 2024

VIKTORVAV99 reviewed Feb 20, 2024

View reviewed changes

PR update

820e2a2

ghost requested a review from VIKTORVAV99 February 21, 2024 12:17

VIKTORVAV99 requested changes Feb 21, 2024

View reviewed changes

VIKTORVAV99 reviewed Feb 21, 2024

View reviewed changes

parsers/ELEXON.py Show resolved Hide resolved

ghost marked this pull request as draft February 22, 2024 10:04

Mathilde Daugy added 2 commits March 27, 2024 15:34

add validation in fetch_production

7ab4d30

update parse_datetime

7116654

ghost marked this pull request as ready for review March 27, 2024 14:49

remove breakpojnt

1c36db7

Merge branch 'master' into md/update-elexon-parser

9c271dc

ghost requested a review from VIKTORVAV99 March 27, 2024 15:18

VIKTORVAV99 reviewed Mar 27, 2024

View reviewed changes

q-- mentioned this pull request May 14, 2024

[Data Issue]: bmreports closedown #6751

Closed

VIKTORVAV99 self-assigned this May 22, 2024

VIKTORVAV99 added 5 commits May 22, 2024 19:53

Merge remote-tracking branch 'origin/master' into md/update-elexon-pa…

d99cba9

…rser

fix any types

127efa3

correct negative with 0

ceb40c1

only merge matching timestamps

ac22fd1

call to_string() in last function

e20d395

VIKTORVAV99 approved these changes May 22, 2024

View reviewed changes

VIKTORVAV99 requested review from madsnedergaard and tonypls May 29, 2024 12:46

madsnedergaard approved these changes May 29, 2024

View reviewed changes

Merge branch 'master' into md/update-elexon-parser

c50947d

VIKTORVAV99 enabled auto-merge (squash) May 29, 2024 14:15

Merge branch 'master' into md/update-elexon-parser

284f76b

VIKTORVAV99 merged commit 51c3210 into master Jun 3, 2024
19 checks passed

VIKTORVAV99 deleted the md/update-elexon-parser branch June 3, 2024 06:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update(parser) - ELEXON #6441

update(parser) - ELEXON #6441

ghost commented Feb 2, 2024 •

edited by ghost

VIKTORVAV99 commented Feb 7, 2024 •

edited

ghost commented Feb 7, 2024

VIKTORVAV99 left a comment

VIKTORVAV99 Feb 20, 2024

ghost commented Feb 21, 2024

VIKTORVAV99 commented Feb 21, 2024

VIKTORVAV99 left a comment

VIKTORVAV99 Feb 21, 2024

ghost Mar 27, 2024

q-- commented Feb 21, 2024 •

edited

ghost commented Feb 22, 2024

ghost commented Mar 27, 2024

ghost commented Mar 27, 2024 •

edited by ghost

VIKTORVAV99 left a comment

VIKTORVAV99 Mar 27, 2024

VIKTORVAV99 May 22, 2024

VIKTORVAV99 left a comment

VIKTORVAV99 commented May 29, 2024

madsnedergaard left a comment •

edited

update(parser) - ELEXON #6441

update(parser) - ELEXON #6441

Conversation

ghost commented Feb 2, 2024 • edited by ghost

Issue

Description

Preview

Double check

VIKTORVAV99 commented Feb 7, 2024 • edited

ghost commented Feb 7, 2024

VIKTORVAV99 left a comment

Choose a reason for hiding this comment

VIKTORVAV99 Feb 20, 2024

Choose a reason for hiding this comment

ghost commented Feb 21, 2024

VIKTORVAV99 commented Feb 21, 2024

VIKTORVAV99 left a comment

Choose a reason for hiding this comment

VIKTORVAV99 Feb 21, 2024

Choose a reason for hiding this comment

ghost Mar 27, 2024

Choose a reason for hiding this comment

q-- commented Feb 21, 2024 • edited

ghost commented Feb 22, 2024

ghost commented Mar 27, 2024

ghost commented Mar 27, 2024 • edited by ghost

VIKTORVAV99 left a comment

Choose a reason for hiding this comment

VIKTORVAV99 Mar 27, 2024

Choose a reason for hiding this comment

VIKTORVAV99 May 22, 2024

Choose a reason for hiding this comment

VIKTORVAV99 left a comment

Choose a reason for hiding this comment

VIKTORVAV99 commented May 29, 2024

madsnedergaard left a comment • edited

Choose a reason for hiding this comment

ghost commented Feb 2, 2024 •

edited by ghost

VIKTORVAV99 commented Feb 7, 2024 •

edited

q-- commented Feb 21, 2024 •

edited

ghost commented Mar 27, 2024 •

edited by ghost

madsnedergaard left a comment •

edited