STORY
Satoshi Settlers - Devlog - Release 0.0.86
AUTHOR
Michael TidwellMichael Tidwell
Joined 2024.01.21
PROJECT
Satoshi SettlersSatoshi Settlers
DATE
VOTES
sats
COMMENTS
NOSTR
View on Nostr ↗
TAGS
#webLN#LNURL#activity#nostr

Satoshi Settlers - Devlog - Release 0.0.86

I've gained significant insights into web browser storage over the past few weeks. A challenge I faced at satoshisettlers.com was that every time a user visited the site and loaded the game, the data wasn't stored in their web browser. Consequently, if they logged out or closed and reopened their browser, they would have to reload the entire game content. To address this, I turned to local browser storage, which typically offers 5-10 MB of storage capacity, depending on the browser's configuration. Now, when users load the game, it first checks the local browser storage for data before reaching out to the satoshisettlers server. This approach not only reduces bandwidth for both the server and the client but also significantly improves load times and initialization. Additionally, the game will automatically update the browser storage if it detects that it is out of sync for more than 15 minutes. This approach ensures that it does not excessively perform I/O operations in your browser but maintains a healthy checkpoint. As a result, there is no need to load numerous blocks if you keep the game open for an extended period.

In this post I will focus on one of the features of this release (The browser compression), but I'll also leave release notes at the bottom, I will discuss how I managed to compress 385 test blocks to the smallest possible size, bearing in mind they still need to be stored as string/text values. This effort was crucial in making the most out of the available storage space, ensuring faster data retrieval, and providing a seamless gaming experience for our users.

Considering the limited storage capacity and aiming for efficiency, I've also taken steps to pre-optimize this solution to extend through all of 2024—at least, that's the plan.
2 compression algorithms I decided to try. Pako and LZ.
385 blocks of json raw takes up 219 KB. If Satoshi Settlers takes off I would soon deplete the local browser storage maximum. So, here are the results of compressing my json.

We can see Pako's compression was about 50% and LZ was about 90%. The reason for Pako not being as good is because PAKO attempts to compress into binary, but localstorage must hold key value strings, so many of the benefits of Pako are not seen here.
Pako was much harder to use than LZ, because of the need to convert my strings into utf8 byte arrays before I could compress.

So I wanted to see if I could do better and combine both Pako and LZ, To do this I first compress to a byte array. But LZ can only compress Strings! So, I need to then base64 encode the byte array. When you base64 encode something it increases in size by 33%, but then we'll be able to compress with LZ, will it be worth it? Let's find out.

Yes, it was worth it, we got an even smaller compression with the combination of Pako and LZ even given that we needed to base64 encode the output of the Pako compression.

But can we do even better? There are many fields in the json payload of a block in the game that can be derived and reduced. For instance the "cost" will always be double the value! There's no need to store this, especially when space is so limited. In the future if there is more complex logic on cost compared to value I will plan to still derive this value. So we can remove cost from our JSON storage!

Next we can see hash (currently not used but will in the future), land index, and resource all stored. However the plan is to utilize the block hash to determine the attributes of the block. All of these can be derived from the hash! Let's remove these additional unnecessary fields.

Here's another field to remove, Because I'm storing the key of the object as the block height there is no need to also store this as a field, we can remove this from our json!

Another optimization is get rid of this big color value I'm storing here as a string. Everything will always have an alpha of 1.0, so I don't need to store this. And it would be much better to not store the native stringified Color struct as json. Instead I'll just store the equivalent 6char hex.

The game engine I'm using doesn't support converting a Color to a hex value, so I made this one :)

Can we squeeze even more space out of this json string storage? I decided to go for it and make all the field keys 1 character. Also made this conversion function that takes a block in game and then converts it to be ready for browser storage.

After all these optimizations let's see where we are at now:

So, we started 219 KB of raw JSON data for our 385 blocks, and we got to 7.88 KB!

This means we can now store 100,000s of game bitcoin blocks in the browser's local storage without hitting maximum limits.

Since the size reduction was quite significant I started thinking about additional fields that would be needed long term like a timestamp. The total size of the 385 blocks came to 11.88 KB! I was very happy as I'm able to also decompress magically and everything works.

Additional Release Notes for v0.0.86

UPDATES RELEASE 0.0.86:

  • bugfix: toggle color button being initiated incorrectly.

  • bugfix: double title on logout page.

  • website feature: nav bar has black background now to help with visibility

  • website feature: debug page now added with link in the nav bar to help troubleshoot user devices.

  • website feature: Web GPU, WebGL, Checkpoint, and Map Size all added to the Debug/ "Report a Problem" page.

  • feature: right click drag to move works now as well.

  • feature: hide text from screen if you zoom out far to help with performance.

  • feature: exit (x) button on the top right of the buy menu.

  • feature: made back button red so it's more noticeable.

  • feature: Made clear button more prominent by making it red.

  • feature: Browser local storage now live and should be helping reduce load time and bandwith of both servers and clients.

  • upgrade maintenance: - Qr img code with update of latest commit on main.

  • open github issue: https://github.com/bevyengine/bevy/issues/11613 (Only can reproduce on Brave)