Repository navigation
Loading cached metadata takes time #1399
Description
My tools are attempting to use the cached metadata from XrmToolBox, but usually, after a while, they start to take a long time to open.
I have tried to delete the metadata folder to restart it from scratch, and it may be a bit faster from the beginning, but it soon becomes slower and slower...
I've tried to find more details about when and why this issue occurred, but I failed. Unfortunately, I'm getting more and more issues from my users about the long waiting...
I believe this cache was created by @MarkMpn a few years ago?
Activity
rappen commentedon Jul 9, 2025
Recent refs:
rappen/BulkDataUpdater#259
rappen/FetchXMLBuilder#1131
MscrmTools commentedon Jul 10, 2025
Yes, this is @MarkMpn code. I must admit I don't know this feature well enough to debug it
MarkMpn commentedon Jul 10, 2025
I'm not sure why it would get slower over time, that's not something I've noticed.
The idea of the cache is that each tool can use it without having to wait to load its own copy of the metadata, so if you're using a few different tools it should be quicker overall. However, as the cache loads the entire metadata in one go it will initially be slower than loading only the metadata for a few tables.
The main process the cache goes through when you connect to an org is:
In orgs where the metadata is changing frequently, it will end up going through the slower path to refresh the metadata more frequently, so maybe it is this that you are experiencing when it appears to get slower over time.
In these cases, most of the work in deserializing the cached metadata is actually wasted, as that cache is thrown away to be replaced by the new metadata. It should be possible to save that time by storing the metadata timestamps separately from the metadata itself so if it is out of date it doesn't need to spend the time deserializing the stored cache.
It's also possible for tools to avoid waiting for the cache to fully load - if you need some metadata while the cache is still loading you can request it directly from the org in the normal way, but use the cached version if it is available. As an example, in SQL 4 CDS I use the
SharedMetadataCache.CacheReadyproperty to decide whether to access the metadata from the shared cache or not, so it should never wait for it to complete.rappen commentedon Jul 10, 2025
Thanks @MarkMpn,
I'm guessing wildly here, but I believe that (in my tools) the cache is always reloaded from metadata, may be that my code is stupid, or even more wildly - since it is checking timestamps, may it always reload since I'm not in UTC, so there's always a diff in time?
But still, "we believe" that it is getting slower over time, like slower now than it was in April etc.
Deleting the whole folder for metadata, it may be faster now, but quickly getting slower...
Codewice, I see that it almost (?) get stuck on line 49 here for 10 sec if we're lucky, but usually half to several minutes...
I guess this happens the first time I start a tool in this connection after starting XTB, so probably it is ok that it needs to reload it.
Just restarting the tool, and it gets really fast.
Starting tools back and forth of connections (but still I the same session of XTB) it gets back to the same panel, but a lot faster than the first time.
I assume my Q is that why can it take minutes?
And why does it need to reload it, even if nothing has happened in the metadata of the connection since last time it was used in XTB? Is my logic a bit dumb?
MarkMpn commentedon Jul 10, 2025
Even if nothing has changed, just deserializing the metadata can take a while because it's so big. It might be that there is a faster way to serialize/deserialize it - I've tried the DataContractSerializer originally but ended up going with Newtonsoft, but there could be a better option.
You'd need to step into the code of the cache loader to see if it's having to refresh the metadata from the server or if it uses the locally saved version though. It doesn't use the actual time to determine if there's a change - we get a token from the server that indicates the metadata version and pass that back to the server so it can tell us if there are any changes, so it shouldn't be a time zone issue.
You could change your logic to only use the cache if it is already fully loaded, like the
CacheReadyproperty in my example above, rather than waiting for it if it is still loading.rappen commentedon Jul 11, 2025
Even my tiniest environment (2 custom tables, a few custom attributes) takes 30+ sec...
I have this option in FXB at least, to not wait for the metadata to be loaded entirely. Although until it's done, you are on your own.
That could be handled better, I guess.
Oh wait... I always have the XTB Storage folder on a OneDrive folder... I guess that might affect.
But I don't know about the users. Hm.
Anyway, I can't find a pattern. Now when I started from scratch, it took sex seconds to load metadata in the "bit" environment. Still on onedrive. It seems like today everything is fast. Because it's an odd date? Even dates the universe is slow? 🤯
rappen commentedon Jul 16, 2025
Moved the storage folder to
C:\XrmToolBox, so now it is only locally on my machine.The first time of the day, it took ~90 seconds to reload metadata. After that, it takes a few seconds.
I still feel that 90 seconds is a long time. Is it reasonable?
This environment has Sales and ~30 custom tables. That's not much...
No changes from us since yesterday, so why does it reload everything again?
Or could it be due to MS updates?
Or is there some timeout in the cache?
Any clues?
MarkMpn commentedon Jul 16, 2025
I can't reproduce this sort of delay on my environment, so I think you'll need to step through the code in
ConnectionDetail.UpdateMetadataCachewith your own environment to see where the time is being spent. Going through that in my environment with an old version of the cache on disk, this takes:So in total this took about 45 seconds for me. Ideally we could shortcut the initial 11 seconds if we could know that it is out of date which would leave us with the remaining 33 seconds, which if we don't already have a cache available is an immovable amount of time as this is dictated by the server.
If you go through this process with a recent version of the cache already available on disk it should skip step 3 and return the cached metadata - in my example here this should take around 12 seconds for the initial load, and then be immediately available for any further requests in the same XTB session.