How We Solved RAM Consumption in EOS Rate

We’ve documented our experience improving RAM consumption in the open-source dApp EOS Rate that utilizes EOSIO blockchain technology on the EOS mainnet.

Before the Update

We recently enabled Eden members to submit ratings on EOS Rate to make the dapp more collaborative and trustworthy for the Eden community. EOS Rate is an open-source project where EOS and Eden token holders can rate block producers in these blockchains based on five categories: community, development, infrastructure, transparency, and trustiness. EOS Rate gathers ratings to help capture a global collective opinion, or “wisdom of the crowd,” and measure if voting patterns truly reflect voter sentiment.

Over the past weeks, we added new features such as visualizing averages for Eden ratings and improved the UI experience overall. However, when we integrated those changes into the smart contract logic, we failed to consider how expensive RAM would become to submit ratings. So, for that reason, and thanks to our community reporting the issue, we worked on solving consumption issues.

After the Eden member implementation, the EOS Rate app works with two main tables: ratings and stats. These tables are scoped by rateproducer and eden. Simple actions executed by the user, known as emplace and modify, define the logic behind the table’s struct as:

Ratings struct (rateproducer, eden):

Stats struct (rateproducer, eden):

The tables shown above calculate the total cost of storing space for the tables identified as ratings and stats at 92 and 36 bytes, respectively. However, we must also consider the consumption from other tables’ fields such as metadata, secondary indexes, and the information that will be persisted. Hence, the final RAM consumption at this point is:

It is evident that the RAM consumption is expensive for this particular case. However, after receiving some reports from users about the reason for the high use of RAM, we became aware of other issues that need to be addressed:

Critical Improvements

1. EOS Rate was one of our first open-source projects when we started on the EOS blockchain back in September 2018. Later, we became aware that we misused the variable types ratings struct, which only stored values from 0 to 10; however, we incorrectly represented those values as float types. Even though it didn’t mean a significant improvement in RAM consumption, we believe that small changes can substantially impact it.
2. Once we deployed the Eden implementation, we decided to scope all Eden and non-Eden members into one single scope — which we called rateproducer. We also added another more specific scope just for Eden members called eden. This way, Eden members will be able to add a record to the general rateproducer and eden scopes on the ratings and stats tables. Before this adjustment, Eden members had to pay double in bytes consumption compared to other EOS Rate users, meaning duplication of the value required.
3. Secondary indexes require high use of RAM consumption with a total of 128 bytes per 64-bit integer key secondary index and double-precision floating-point key secondary index. Previously, on EOS Rate, there was a declared secondary index, but the smart contract logic didn’t yet use it.

During the Update

We aimed to optimize the variable type on ratings and stats struct. Although float is required to store average numbers in each category on stats, variable types can still be optimized on ratings. Also, the uniq_rating field is needed as part of the EOS Rate logic with no valid reason to persist. However, instead of storing value, we can calculate it every time it’s needed since uniq_rating results from the unit64_t concatenation provided by user and bp name values. The smart contract also uses this value to index it on a specific ratings row to update it. The next step was understanding EOSIO RAM consumption.

How RAM Consumption Works on EOSIO

RAM consumption is more complex than just calculating the sum of the total bytes on one table. It usually uses some backstage logic that needs to be addressed, for example, struct fields, secondary indexes, and table metadata (including instances) with a particular behavior of RAM consumption.

Struct field

Obtaining the struct total bytes is as simple as adding the values on all table types, as shown on the first two tables. For example, if we have a struct of a table with 2 float and unit8_t type, the cost is 9 bytes because float is 32 bits and unit8_t is 8 bits. Then, 64 bits divided by 8 is equal to 8 bytes and 8 bits. The final result then is 8 bytes and 1 byte, which is equivalent to 9 bytes.

Secondary indexes

The behavior of secondary indexes is different from struct fields. Some constant values need to be considered depending on the declared secondary index type. For instance, there is a constant value for:

1. 64-bit integer key secondary index and the double-precision floating-point key secondary index: It may cause an additional 128 bytes of RAM to be billed per table row.
2. 128-bit integer key secondary index and the quadruple-precision floating-point key secondary index: It may cause an additional 144 bytes of RAM to be billed per table row.
3. 256-bit integer key secondary index: It causes an additional 160 bytes of RAM to be billed per table row.

Table Metadata

A Multi-Index table has different states of behavior:

1. It has no row/s (full-emplace): When a user wants to add a new register to an existing Multi-Index table with no rows, the RAM payer needs to pay for extra RAM for the data serialization and the “table metadata” instance that has a static RAM cost of 112 bytes + table structure, which is another 112 bytes.
2. It has row/s (emplace): It happens when there already exists at least a record on the table. In that case, a new emplace will pay only 112 bytes for “table metadata” + serialized struct.
3. It has row/s (modify): When there already exists at least one record on the table, and a user wants to modify a specific record, the RAM payer may change. However, the RAM cost is going to be the same as first the emplace (emplace).

How RAM Consumption Worked on the rateproducer Smart Contract

The rateproducer smart contract has two tables: ratings and stats. Both tables are scoped by rateproducer and eden. When a non-Eden user submits a rating, an emplace action is executed, generating a RAM cost of 720 bytes. On the other hand, when an Eden user rates, the action takes 720 bytes more RAM than a regular user because the Eden users were doing four emplaces; two emplaces for our general scope, rateproducer, and the other two emplaces for our specific scope. In this case, eden scopes while non-eden users were only doing two emplaces considering both tables. You can view the final result of the RAM consumption calculation in the following table:

After the Update

Eden members may experience a significant RAM consumption improvement with almost 73% fewer bytes required per rate. Non-Eden members may also notice an improvement in RAM of nearly 30% fewer bytes per rate.

What We Did

1. We separate the logic on our staging branch by querying it to the genesdeden account on Jungle 3 (which is an account that emulates the official Eden members table) and our master branch to genesis.eden account on Mainnet.
2. We generated the abi file with its Ricardian Contract and Clauses from CLSDK.
3. We removed the uniq_rating field from the new ratings_v2 struct. Instead of storing uniq_rating we now calculate it every time it is needed.
4. We removed an unused secondary index by_user.
5. We created a migration function to move data from the ratings to the rating table. At the same time, it will update the stats data on the rateproducer scope by removing eden member rates.
6. We updated the table scopes. Now, Eden members will emplace only on the rating table under the eden scope. To calculate the non-eden and eden average, we need to apply the following formula:

Things We Will Do

We plan to make a secure migration to preserve the integrity of the data that is stored on the rateproducer account. To achieve this, we will first start to migrate the data from ratings to rating scoped by rateproducer and eden. At the same time, the migration action will update the stats content by removing from the rateproducer stats scope the eden ratings. The eden scope data on stats will not need to be updated because that scope only considers Eden ratings.

After the first migration phase is complete and successful, a second migration will free up the RAM of the old table struct for all the rates made.

Overall bytes improvements

With this improvement in RAM performance, we understood RAM cost better to avoid the same issue persisting in the future. We are also happy with the resulting outcome of 71.67% for Eden members and 26.88% for non-Eden members. EOS Rate users can now rate without the fear of running out of RAM. RAM consumption is now more passive, meaning Eden members will notice an improvement when rating and not be affected by our method to prevent Sybil attacks. We trust Eden members will not try to harm the transparency purpose of EOS Rate.

Written by: Leister Álvarez.

EOS Costa Rica operates since 2018. Our team develops blockchain-based solutions with great attention to detail in creating user-centered dapps for enterprise use. We also provide resources to the EOSIO infrastructure and promote the local EOSIO ecosystem. Let’s talk about how you can implement this technology into your organization.

Follow us on social media:

• Website: https://eoscostarica.io
• Twitter: https://twitter.com/eoscostarica
• Telegram: https://t.me/eoscr
• Facebook: https://www.facebook.com/costaricaeos
• LinkedIn: https://www.linkedin.com/company/eoscostarica