wild numbers after a panel reset

[sargonas]

So I've been tracking a strange issue for a while that I was just self correcting when it happened and dismissing as "a problem for later" but today I gathered enough info from what happened to bother documenting things, and was also able to tie it to a cause finally: a panel software update.

Every now and then, my panel will go off the charts with one single reading, like one incremental reading of all sensors will be so insanely large, it will be more in that one momentary reading than several days or weeks of data. I've now come to realize, this most commonly happens when the panel "reboots" after a software version update (though I don't yet think it is exclusive to this).

Here are my charts for today with this happening, and here is some of the raw data examples (these charts are not flat lined, they are quite vibrant, but this ONE data point is so off the charts, pun not intended, that it flattens out all the other data points):

Everything seems to just be... multiplied by a factor of "holy crap" or something?

This one is an exceptionally curious one, because my kitchen vent hood fan is the ONLY thing on that circuit and was never turned on today, because I'm not even at the house today!

So this is weird, it seems when the system restarts, from a software update or else wise, some weird data is getting into the system. I also need to collect more number in details, but sometimes for some circuits, it seems like they have the exact same aberrant numbers shared between them at times.

For a while now I have been popping open the developer tools and hand editing the data down to 0 for these readings for all 20 some-odd devices when this happens, but truth be told doing this once a month or so is getting old and I finally have time to dig into it deep and help get to the source of this. Would love to see if we can nip it in the bud, either because we're doing something goofy with aberrant data from the panel, or if the panel is just being wonky and we can somehow monitor for and avoid capturing that?

I need to pull detailed numbers in the morning, but my initial suspicion is that upon certain panel restart states, it's polling the "current" total increasing number as a fresh count with a delta from 0 maybe?

This time around I'm going to wait to touch any data, so I can pull anything relevant that might help. (Oh, and I am running v 1.0.10, but this has been going on for as long as I can remember, fwiw)

[cayossarian]

Yeah, I never really noticed it but now that I look back I can seem some strange data like -9000kWh in untracked data and some other oddities that are clearly out of line.

Since the the integration itself is python it's not likely to produce erroneous data on its own (variables are automatically initialized) but I if the panel firmware is written in some compiled language with uninitialized variables data could be virtually anything until those variables are explicitly set. Also If the network were to have a hiccup this could also introduce some errors. Ideally there would be a checksum to ensure integrity but there isn't.

We could introduce some threshold, for instance some Kalman filter or the like, where data that is clearly off base could be discarded. There really would be little harm in tossing 15 seconds worth of data evert once in a while and certainly detecting it is useful.

[sargonas]

I was thinking something similar. Currently doing the long drive from LA back to Vegas, but when I get back to my house over the next day or two I was gonna pick through some of the code and try to look and see if there wouldn't just be a super easy way for me to pop in a check that basically if a reading is bigger than a specific percentage from the most recent non zero reading that's somewhat sane and sensible, it will discard that particular reading and wait for the next reading.

I still need to sit down and pull logs and do some comparisons and try to manually trigger the state on my panel myself (that might just require I kill the master power for a second or two… No big deal lol) but I'm getting more and more suspicious that on the very first API read after a reset it's just returning The entire sum total of that sensor in a way that looks like a fresh reading.

Also you mentioned untracked data and that's been an ongoing headache of mine for months. I keep getting wild variations of untracked data, sometimes small amounts sometimes very not trivial amounts. The thing is it's not possible for me to have any power going out or coming in that is untracked yet there it is on some days… Quite strange.

[cayossarian]

I am in the process of adding a feature set that allows one to create sensors or attributes with user defined mathematical states and actions. The original use case was to allow a user to set a threshold for a sensor or to create new sensors in a hierarchical fashion. The possibilities are really open ended.

This issue you raised fits like a glove because one could define an sensor attribute with a range where the sensor reading could be discarded and/or another range where the user could be notified (load exceeds spec).

The basic underpinnings for math awareness is in place (I use it to add two 120V circuits together to show solar in a single 240V sensor).

My next step is to replace all the sensors with these math aware sensors and then some configuration UI.

I'd be interested in any approaches you come up with to address this issue once you dig into the data.

[sargonas]

I think I might have an idea. Looking at how sensor.py is setup, I think I can fix all of this with a simple validation that if a reading is more than x, invalidate it. I am thinking a safe starting point is 20 kWh, because a 60am circuit would pull something like 14.8 or so kWh, which is the biggest circuit I can think of in a home if an EV was charging. I'll share a branch shortly so you can validate my math because I get pretty scattered brained when dealing with Watts vs Watt hours and then kW and kWh etc, lol

[sargonas]

Okay, I pushed my changes to b90ef99 in the new branch reset-values-fix. Would love you eyes on it when you get a chance @cayossarian, because I am pretty sure there is room for improvement in my hacky pass at this. (and I would not take any bets against me on having brain farted a unit conversion somewhere!)

I adjusted the circuit method to use the following thresholds:

self._max_energy_jump_wh = 20000 - defaults at 20 kWh max jump between readings since a 60amp circuit can't pull more than 15 kwh, and I can't think of a residential setup where there would be more than that even on an EV charger.

self._max_energy_rate_w = 50000 - 50 kW max power rate feels very generous to avoid false positives I think? I don't think we even really need to worry too much about the energy one, but I figured while I was in there tinkering may as well.

In theory i focused this at the circuit level only, so I don't THINK it will have any knock-on effects to the solar and grid readings, which are quite higher. The only problem I can see with this is a super slim edge case on if someone has a circuit that is rarely if almost never used, and so its cumulative lifetime hasn't even reached that number yet, so every time it resets it will accept the reading, but I'm not sure how severe of an edge case that is?

(Oh yeah, almost forgot to mention, it also throws a message in the logs if it has to drop values for threshold exceptions)

[cayossarian]

Great, if you're running this let me know your experience. I'm in the process of adapting the sensors as I mentioned so we can use your settings as a proof of concept where the user will be able to set the parameters and the math aware sensors will do the heavy lifting without hard coding.

If we have log entries when the circuit discards data that will tell us a lot about when this scenario is kicking in.

I'll get a beta out with the new sensors soon so don't spend a lot of time tweaking code in your branch.

[sargonas]

Sounds good. I do have it running, I can say everything else is working as expected but since these issues only occurred for me about once or twice a month tops, who knows when it might trigger in order to give me something meaningful. That said I bet I can cause my panel to reset without killing the power to my house (or worst case sure I could just do that too) so maybe I’ll do that later today and just see what happens. (Fortunately my home lab is on a battery back up so none of my infra will go down, which means the panel reset will still be captured by home assistant in real time.)

…

On Thu, Jun 26, 2025 at 08:32 Bill Flood ***@***.***> wrote: *cayossarian* left a comment (SpanPanel/span#87) <#87 (comment)> Great, if you're running this let me know your experience. I'm in the process of adapting the sensors as I mentioned so we can use your settings as a proof of concept where the user will be able to set the parameters and the math aware sensors will do the heavy lifting without hard coding. If we have log entries when the circuit discards data that will tell us a lot about when this scenario is kicking in. I'll get a beta out with the new sensors soon so don't spend a lot of time tweaking code in your branch. — Reply to this email directly, view it on GitHub <#87 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAALDF3E2T3LGFJI75QU6VT3FQHA5AVCNFSM6AAAAAB77HROJKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTAMBYHEYDKMBUGI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[sargonas]

So, Span pushed another firmware update to me today!

Good news: I was able to catch it doing it's thing in real time and was able to confirm 100% that yes, it's reading the full true API value and not the delta upon a reboot of the controller software powering the panel's brains.

Bad news: While I thought I was test running my code, turns out... I wasn't. Oops. So it still borked and I didn't get to test the fix. Working on that now but I don't expect them to push a change any time soon, so I guess I'll have to force a reset instead once I do. XD

However this opens a new can of worms that I think brings my fix into question. I cross verified these things by hitting the API directly in terminal and the API endpoint for circuits always returns the full value. This makes me think there is logic somewhere else in the span code that is distilling the value read down from the full value to the delta... however, I can't find that in the circuit code that I modified, which means this logic would be happening elsewhere in the code and presumably after the API call in the file I changed? If after, then what is it using as the key reason to NOT extract a delta this time? Is it because a delta read against an unavailable call vs a delta read against a 0 or other value call is the triggering issue? If so then that should be where I dig in for the fix maybe.

Actually this tracks as I write this out. notice how many of our reads are 0 in the developer tools data panel? Those api end points NEVER return a 0... but I think the original span code before we started playing with it is reading the api, and then doing the delta conversion itself? I'm going to dig deeper, I'm increasingly more and more convinced there is logic elsewhere handling this that needs to be changed, not the code I added (not to mention that logic probably will be easier to shore up for this as well.)

[sargonas]

Okay I gave down down the rabbit hole of how HA processes sensor data. It seems HA itself is doing the logic on taking an API call and then delta-ing that sensor data and updating the energy data points from it. This is just part of how Energy sensors work in HA.

So now the question is, what does HA use as it's "Tracker" to know when a read should be delatad, and what against? Whatever logic that is, likely exists as a standard way that other integrations use, and just needs to be added in here maybe?

[sargonas]

Sorry for the stream of consciousness posts @cayossarian!! But I think it figured it out.

When the panel goes offline, we return the sensors as Unavailable since the panel is 502ing to us. When the panel comes back online, we return the values again. HA's internal logic looks a the data reads and goes Unavailable --> 123456wh = a delta of 123wh and writes a sensor update of 123456wh.

So this is, technically, internal logical behavior by HA that is mostly intended, so we just need to code around it on our side. My current suggestion is that when we can't contact the panel, we keep the sensors available by simply returning the same previously known good values we had until it comes back online, or at least for a semi reasonable amount of time, maybe 10 minutes or so (all resets seem to be less time than that) to account for intentional resets but leave an off ramp for long term major outages.

Edit: I've gone ahead and made some really simple changes to do this and rolled back my previous change and added these in to my branch.

basically in the coordinator now there is a last_update_success method that is tracking successful reads to the panel. If it can no longer reach the panel, for 15 minutes it will keep returning the old values. I then created a second available property that, instead of inheriting the base one from the other class, is used specifically for this use case that uses that last_update_success as a guidance to know if it should return available or not, and if so what to return.

these changes are in my branch for you to review and pick apart, and I'll try to to test them on my side in the meantime, once I can figure out how to get HA to happily load a specific test version without wiping out my integration.

[cayossarian]

to my knowledge the integration is not doing any delta conversion. The energy is the full value and supposed to be always increasing.

HA is of course showing kWh instead of Wh so it's doing some work there between time intervals. If several readings arrive and one is huge the result will be the average in that interval.

I'm closing in on my changes which should help us out

[cayossarian]

Your analysis makes perfect sense for energy. The integration does not return zero if the panel is not returning data it tells HA that the update fails but only after it retries a few times and then it returns a fail to update. The math I mentioned in HA to get to kWh may result in what you're seeing.

The fix for energy may be to return the same energy value for a period if your suspicion is correct but for power it would need to return UKNOWN for every circuit including panel level main, status, etc. It's worth a try.

[sargonas]

Yup, that's what these new changes in my branch do exactly. Energy stats (the Wh endpoints) for each circuit now returns the last known good value for up to 15 minutes after the panel is first seen to be offline/unreachable, but all other endpoints will return Unavailable still. After 15 minutes, if the panel is STILL unreachable, Energy will also return Unavailable as well. That's pretty much the only change this does, and that 15 minutes was something I picked arbitrarily and can be adjusted as well if need be.

[cayossarian]

1.2.2 addresses, let me know if you see any more problems and thanks for digging into it.