April 2024 (starting) enhancements to ‘Trends Engine’ inside of ez2view, in preparation for IESS

On Thursday 17th April 2024 we published an initial version of this Release Note  (pertaining to enhancements made to the ‘Trends Engine’ component within ez2view).   We published that note on that day because ez2view licensed users using certain data series in the ‘Trends Engine’ from that day will have started encountering a Warning Message looking like the following:

Since that time, we’ve become aware that sufficient extensions would be beneficial to that earlier note that it was thought that:

1)  it would be more appropriate to re-publish as this new article; and

2)  we’d update the context of the earlier article to point to this one to give clients the fuller picture.


Accessing the ez2view ‘Trends Engine’ through both Access Points

Remember that there are two different access points for clients who have a licence to ez2view:

1)  Much of the value being via the installed software;

2)  But with also a small number of widgets accessible online via your browser of choice in ez2view Online.

The ‘Trends Engine’ component in ez2view is accessible through both access points … though the functionality is slightly different (necessarily) in each case.


What is IESS, and why is it relevant to ez2view?

Many of our clients will have been aware that our development team has been investing heavily through the first half of 2024 to prepare various products (including ez2view) for the  looming IESS market changes.  We’ve already provided several updates on this Timeline site:

1)  firstly on 9th January 2024 with ‘Preparing ez2view for IESS (IRP and BDU), which goes live Monday 3rd June 2024’,

2)  then in this ‘Progress report, in upgrading ez2view for IESS’ on 3rd April 2024,

3)  and (specifically with respect to the ‘Trends Engine’ within ez2view) via Dan’s article from 17th April, which is being superseded with today’s more detailed explanation.

However before going into the detail, we recommend you read the background note ‘What is IESS, and why is it relevant to ez2view?’ that’s been published at the same time, and is intended to explain:

  • More about the IESS change itself;
  • and, importantly, why it is relevant to ez2view licensed users.

Assuming you have read this linked article first, then read on further below …


What enhancements have been made to ‘Trends Engine’ within ez2view?

In this short release note, we would like to highlight specific enhancements that we have begun to implement inside the ‘Trends Engine’ within ez2view, with more to come in the near future.  These still-to-be-delivered changes will follow the same pattern as below:

Enhancement #1 = We’ve started to change the tab structure

There used to be a tab named ‘Generators’ featured prominently in ez2view, but this was becoming both:

Problem 1)   increasingly misleading … in that the title was ‘Generators’ and yet it already contained similar information for ‘Loads’ and would have been required to also contain ‘Bi-Directional Units’ with the IESS changes …

…  which (e.g. when coupled with various permutations of the ‘Aggregation’ function – i.e. selecting one of the circles to sum everything nested below) ran an increasing risk of returning data that was at odds with what a user might expect, were they not aware of these quirks;

Problem 2)  Coupled with this, the number of data series had continued to grow on some tabs, making individual series more difficult to find and/or understand.

We have grappled with these challenges (amongst others) and so have started to implement …

The Solution = as a result of which, we’re rationalizing the tab structure in ‘Trends Engine’ inside of ez2view, to assist in making data easier to find.

With respect to this image, note the following:

Enhancement #1a = New tabs being added

You’ll see that the first batch of enhancements (on Thursday 18th April 2024) included the addition of a new ‘Units (NEM)’ tab – which has been designed to make it easier for clients to find and select the data they are looking for.

Other tabs to be added soon include:

i.  A tab containing NEM-specific units that participate in any of the 10 x FCAS commodities, and data relevant to that
ii.  A tab containing WEM-specific units (i.e. for the WA market of the South-West Interconnected System).

… for now, you can still find those data points on the ‘Generators (legacy)’ tab.

Enhancement #1b = Generators (legacy)

As an interim step in this process, we have moved the previous ‘Generators’ tab to the end of the menu bar and labelled it as ‘legacy’.

We envisage that this tab will eventually be removed … but only after all the data series you need access to are provided for on more clearly named new tabs.  Users who still want to access data series on this ‘legacy’ tab should note that some of the data series on this tab have been changed, as discussed further below.


Enhancement #2 = Deploying a uniform number sign

As described in the related article ‘What is IESS, and why is it relevant to ez2view?’, we have invested time to shift all data sets included in ez2view (i.e. across all widgets) follow this simple rule:

Positive numbers = represent injections onto the grid (i.e. discharge from a battery, discharge from hydro, generation from coal, gas, wind, solar, etc…).

… remembering Tripwire #3 (introduced with WDRM), this will continue to mean that dispatch of Negawatts will be seen as a positive number (i.e. a supply-side quantity, even if intangible).

Negative numbers = represent withdrawals from the grid (i.e. charging a battery, pumping uphill to a pumped storage reservoir, and consumption by other Scheduled Loads).

It’s worth stating explicitly, then, for any client who also queries the EMMS directly, that:

1)   For those units that will be re-registered as a ‘Bi-Directional Unit’ (BDU) after 3rd June 2024, these values will be positive when generating and negative when charging (which is .

2)  However the number sign may on (special corner-case) occasions be the opposite to how data is represented in the EMMS.

(a)  A case in point will be for Scheduled Loads (which in the EMMS are still positive if consuming, but in ez2view will be negative);

(b)  Though note for Negawatts, because of Tripwire #3.


Enhancement #3 = Shifting to ‘end-of-interval’ values wherever possible

When we first provided access to this ‘Trends Engine’ application within ez2view, one of the principally used data points was ‘Initial MW’ for each DUID.

(a)  This was the SCADA snapshot of the units output (or consumption, if a load) at the start of the Dispatch Interval that the time point represented

(b)  But remember that AEMO standard timing convention is that dispatch intervals are time-stamped with the end-of-interval time point … hence this tended to cause confusion with some clients.

Many months ago as an enhancement to ‘Trends Engine’ within ez2view we added in ‘Final MW’ as a value that time-shifted the InitialMW values to be end-of-interval values.

… we did not note in specific a Release Note at that time.

With the recent changes (April 2024) are striving, where possible, to ensure all data is end-of-interval basis.


Enhancement #4 = Improving (clarifying) the data available

At this point, it’s useful to remind readers about this common ‘Analytical challenge (or beginner mistake!) – understanding the difference between a MW and a MWh’ that was published as a discrete article on WattClarity in October 2022…

…  this short article was extracted out of a longer explanation of the gory details of ‘measuring’ demand in the NEM that was published back in 2018.

With IESS prompting the need to simplify the data included in ‘Trends Engine’ within ez2view moving forwards,  we have taken the initiative to invest further time to simplify

Enhancement #4a = new (clarified) data series on the ‘Units (NEM)’ tab

On this new ‘Units (NEM)’ tab, we have separated off data sets that do not pertain to supply/consumption of ENERGY in Australia’s National Electricity Market (NEM) and stripped back initially to just 9 x Data Series as follows:

This number is intentionally small, to assist clients in finding/selecting the data series they are looking for.

There are a couple more series that will (most likely) be added back into this tab (after being labelled more clearly), but we’re not planning on the tab containing many more data sets than that.

Note that we don’t yet have a Glossary on the ez2view Help Site that might help users to understand the data that they are dealing within the ‘Trends Engine’ with respect to the data series the are now available on this tab.  In the absence of such a Glossary, here’s a short note about each of them:

Energy (MWh)

This is a data series that’s a volumetric measure (i.e. MWh) for each dispatch interval that is derived from linear interpolation between two different ‘Power (MW)’ measurements at the start, and the end, of that dispatch interval.

With the switch to bi-directionality with the number sign:

1)  a positive number is energy injected onto the grid and a negative number is energy withdrawn (i.e. consumed) from the grid for that dispatch interval.

2)  hence, aggregating across units (and/or across time) will yield a positive or negative number that means the same in aggregate.

So for those who want to identify total ‘generation’ (i.e. in terms of injections onto the grid), be careful in selecting the data series you want to aggregate across.

Power (MW)

This is an instantaneous (SCADA-level) metered rate of production (positive number) or consumption (negative number) for the unit at that particular point in time.  Remember that:

1)  the NEM is dispatched on an as-generated basis (i.e. ‘gross’) … and that this data is what’s published publicly by AEMO; whilst

2)  The NEM is settled on a sent-out basis (i.e. net of used-in-system, or auxiliary, load) … but this revenue meter data is confidential to the participant.

Hence in the ‘Trends Engine’, as for the rest of ez2view, all generation data is an ‘as generated’ number (i.e. inclusive of  auxiliary energy used onsite in the production process).

Target (MW)

This is the instantaneous required rate of production that’s calculated within NEMDE at the start of the dispatch interval for the instantaneous level of power output (or consumption, if a Scheduled Load) the unit needs to meet at the end of the dispatch interval.

This data:

(a)  Exists for every dispatch interval for each Scheduled and Semi-Scheduled unit (though note that the Target is more like a Cap for Semi-Scheduled units)
(b)   But does not exist for Non-Scheduled units (except in some particular cases of Non-Scheduled units with Semi-Scheduled requirements).

Capacity (Generation) and Capacity (Load)

These are the maximum potential to inject onto the grid (generation, a positive number or zero) or consume from the grid (load, a negative number or zero), as noted in the registration of the given unit.

… these are based on what are defined as ‘Maximum Capacity’ and ‘Minimum Capacity’ in the EMMS v5.3

We have not forgotten what we’ve previously written with respect to ‘Analytical Challenge – choosing what measure to use, for ‘Installed Capacity’, and have not firmly decided what to do in relation to ‘Registered Capacity’ (not currently used in this initial set of data in this new tab).

… if you have particular views, please let us know?

Availability (Generation) and Availability (Load)

These are the maximum potential to inject onto the grid (generation, a positive number or zero) or consume from the grid (load, a negative number or zero), as noted in the registration of the given unit.

Availability (UIGF)

This data is only published for the Semi-Scheduled units.  UIGF stands for Unconstrained Intermittent Generation Forecast, and is the energy-constrained Availability for the unit.

(a)  Linton Corbet’s article on WattClarity ‘What inputs and processes determine a semi-scheduled unit’s availability’ from August 2023 would be useful for readers to refer to here.

i.   to be clear, the number accessed under this series is illustrated as ‘UIGF sent to NEMDE (3.3)’ in this diagram in Linton’s article.

ii.  so it is still subject to any games/optimisations/manipulations* used by self-forecast providers, and we are contemplating whether it is necessary to also include the data stream described as ‘AEMO Forecasts (3.1)’ in this diagram in Linton’s articleLet us know if you have particular requirements?

re (*) different readers will choose their own form of the descriptor here based on their own sense of the validity of the (e.g. biasing) approaches sometimes utilised by some self-forecast providers.

(b)  It’s particularly important to understand that this UIGF number will sometimes be higher than the ‘Availability (Generation)’ number:

i.  following the change enabled for Semi-Scheduled units from 7th August 2023, this will occur on occasions where MaxAvail in the bid has been used to declare some physical limitation at the plant;

ii.  From our GSD2023 product we prepared the article ‘How many Semi-Scheduled units have taken advantage of the market change that went live on 7th Aug 2023?’, looking at the ~5 months to 31st December 2023.

Semi-Dispatch Cap

This data is only published for the Semi-Scheduled units.   The Cap is set when AEMO (via NEMDE) requires the unit to ‘follow’ its Target …

(a)  noting the requirements were tightened from 12th April 2021

(b)  but, even in this case, the Target is more like a Cap than a specific measure to absolutely reach (at least in terms of AEMO Conformance).

Enhancement #4b = changed (clarified) ‘old’ data series on the ‘Generators (Legacy)’ tab

As part of the process of (eventually) deprecating the ‘Generators (Legacy)’ tab, but in the interim period ensuring that it is as correct as possible in relation to Enhancement #2 above (a consistent number sign), we have made some changes/corrections to the data series that were previously available on that tab.

It’s important for users to note that this will have had the effect of changing the output of a user’s pre-configured queries using these series.

… individual clients are urged to speak directly with us if they have concerns about this approach?

With respect to the data series that have changed:

Energy Generated (Estimate)

There are two changes made to this data series:

(a)   We have renamed ‘Energy Generated’ to now be labelled ‘Energy Generated (Estimate)’ to better reflect that this is an estimated value.

(b)  But more importantly, since this ‘Generators’ tab includes all units (and since consuming units do not “generate” energy), we have altered the series to be zero for any unit that is consuming:

i.  this mostly affects Loads but also Bi-Directional Units in the future.

ii.  in other words, it will always be a positive number (i.e. representing injections into the grid).

InitialMW (Generation))

With respect to the (formerly named) ‘InitialMW’ data series :

(a)  This has has been renamed ‘InitialMW (Generation)’  to make it clearer; and importantly

(b)  Has been set as a 0MW data series for any Load.

Additionally, the (formerly known as) ‘FinalMW’ series:

(a)  has been renamed ‘Final MW (Generation)’.

(b) Has been set as a 0MW data series for any Load.

Coupled with the above …

InitialMW (Load)

We have added a new series ‘InitialMW (Load)’ which has is the SCADA snapshot at the start of each dispatch interval for the pre-IESS Loads:

(a)  Excluding Negawatt Units, as they are really supply-side units (despite the confusion of Tripwire #3)

(b)  But (unlike the clearer approach used for the new ‘Power’ series above, where we reverse the sign) we have not done this for InitialMW (Load) because it’s still named as for the field in the EMMS.

Because this is a legacy tab and will be retired in future, we did not implement a ‘FinalMW (Load)’ series.


Enhancement #5 = Simplifying the ‘Group By’ facets

As part of our push to simplify ‘Trends Engine’ inside of ez2view to make life easier for our clients, we’ve simplified the number of facets available to group by (splitting onto separate tabs helps to make this possible).

In particular it’s worth noting 2 of the 6 facets that remain for this tab:

Enhancement #5a = A single ‘Type’

Anticipating the addition of hybrid units registered behind a single DUID (i.e. where production won’t be explicitly coming from a particular fuel type) we have simplified to a singular classifier we’re calling ‘Type’ as follows:

 As shown in the screenshot above, there are currently eight values within this new category. We have implemented this new category in order to make it simpler for users to select aggregations of units, especially for storage units.

Please note that the new ‘Units (NEM)’ tab only includes units registered for the Energy Market. In a future release, we will introduce another new tab focused on the selection of Units registered for FCAS.

Enhancement #5b = Sorting by ‘Dispatch Type’

Where the user wishes to sort/filter further (e.g. by specifically excluding Loads in an aggregation) it would be possible to do this by using the ‘Dispatch Type’ filtering mechanism, as illustrated here:

Note with respect to the above that Bi-Directional Units will have their own (new) Dispatch Type.


What about other widgets within ez2view (the installed software)?

Stay tuned here on this site for further updates in the coming weeks … or contact us directly, if you have specific questions