feat: integrate token list controller storage service #24019

sahar-fehri · 2025-12-15T14:35:36Z

Description

Do not merge until this is released MetaMask/core#7413

Performance Comparison: Per-Chain Token Cache Storage

This PR implements per-chain file storage for tokensChainsCache in TokenListController, replacing the single-file approach. Each chain's token list is now stored in a separate file, reducing write amplification during incremental updates.

📊 Complete Performance Comparison

Cold Restart

Metric	This PR	Main Branch
getAllPersistedState	235ms	288ms
TokenListController read	0.04KB (shell only)	4,102KB
Cache load	97ms (parallel reads)	135ms (single file)
Total overhead	~332ms	~288ms

Main is ~44ms faster on cold restart (single file read vs parallel reads + getAllKeys overhead)

Onboarding

Metric	This PR	Main Branch
Total data written	4,070KB	9,472KB
Number of writes	7 (one per chain)	5 (cumulative rewrites)
Total write time	~38ms	~118ms

This PR writes 57% less data and is 3x faster

Add New Chain (Monad)

Metric	This PR	Main Branch
Data written	33.79KB	4,103KB
Time	0.23ms	45.34ms

This PR is 121x smaller and 197x faster!

Summary

Category	This PR	Main Branch	Winner
Cold restart	~332ms	~288ms	Main (+44ms)
Onboarding writes	4,070KB	9,472KB	This PR (-57%)
Onboarding time	~38ms	~118ms	This PR (3x faster)
Add chain writes	33.79KB	4,103KB	This PR (-99%)
Add chain time	0.23ms	45.34ms	This PR (197x faster)
Write amplification	None	Severe	This PR

📋 Captured Logs

This PR - Cold Restart

[ControllerStorage PERF] getAllPersistedState started
[ControllerStorage PERF] TokenListController - 0.04KB - read: 89.00ms, parse: 0.00ms, total: 89.00ms
[ControllerStorage PERF] getAllPersistedState complete - 235.37ms

[StorageService PERF] getAllKeys TokenListController - 7 keys found - 277.37ms
[StorageService PERF] getItem TokenListController:tokensChainsCache:0xa - 96.19KB - read: 3.12ms, parse: 0.47ms, total: 3.59ms
[StorageService PERF] getItem TokenListController:tokensChainsCache:0x1 - 1608.95KB - read: 30.86ms, parse: 10.57ms, total: 41.43ms
[StorageService PERF] getItem TokenListController:tokensChainsCache:0x38 - 1288.32KB - read: 48.95ms, parse: 21.65ms, total: 70.60ms
[StorageService PERF] getItem TokenListController:tokensChainsCache:0x89 - 324.12KB - read: 72.62ms, parse: 5.21ms, total: 77.83ms
[StorageService PERF] getItem TokenListController:tokensChainsCache:0xa4b1 - 222.52KB - read: 77.90ms, parse: 7.06ms, total: 84.96ms
[StorageService PERF] getItem TokenListController:tokensChainsCache:0xe708 - 46.92KB - read: 85.16ms, parse: 0.82ms, total: 85.97ms
[StorageService PERF] getItem TokenListController:tokensChainsCache:0x2105 - 481.64KB - read: 88.85ms, parse: 8.74ms, total: 97.58ms

This PR - Onboarding

[ControllerStorage PERF] getAllPersistedState complete - 731.91ms

[StorageService PERF] getAllKeys TokenListController - 0 keys found - 309.51ms
[StorageService PERF] getItem TokenListController:tokensChainsCache - NOT FOUND - 33.51ms
[StorageService PERF] setItem TokenListController:tokensChainsCache:0x1 - 1610.14KB - stringify: 8.64ms, write: 8.54ms, total: 17.17ms
[StorageService PERF] setItem TokenListController:tokensChainsCache:0xe708 - 46.92KB - stringify: 0.19ms, write: 0.08ms, total: 0.26ms
[StorageService PERF] setItem TokenListController:tokensChainsCache:0x2105 - 481.34KB - stringify: 1.50ms, write: 2.45ms, total: 3.96ms
[StorageService PERF] setItem TokenListController:tokensChainsCache:0xa4b1 - 222.53KB - stringify: 1.03ms, write: 0.52ms, total: 1.56ms
[StorageService PERF] setItem TokenListController:tokensChainsCache:0x38 - 1288.32KB - stringify: 4.74ms, write: 6.49ms, total: 11.23ms
[StorageService PERF] setItem TokenListController:tokensChainsCache:0xa - 96.19KB - stringify: 0.31ms, write: 0.52ms, total: 0.83ms
[StorageService PERF] setItem TokenListController:tokensChainsCache:0x89 - 324.46KB - stringify: 1.10ms, write: 1.72ms, total: 2.82ms

This PR - Add New Chain (Monad)

[StorageService PERF] setItem TokenListController:tokensChainsCache:0x8f - 33.79KB - stringify: 0.17ms, write: 0.07ms, total: 0.23ms

Main Branch - Cold Restart

[ControllerStorage PERF] getAllPersistedState started
[ControllerStorage PERF] TokenListController - 4102.55KB - read: 112.51ms, parse: 22.77ms, total: 135.27ms
[ControllerStorage PERF] getAllPersistedState complete - 288.21ms

Main Branch - Onboarding

[ControllerStorage PERF] getAllPersistedState complete - 785.03ms

[ControllerStorage PERF] setItem TokenListController - 0.06KB - stringify: 0.00ms, write: 0.02ms, total: 0.02ms
[ControllerStorage PERF] setItem TokenListController - 1609.28KB - stringify: 13.41ms, write: 11.58ms, total: 24.99ms
[ControllerStorage PERF] setItem TokenListController - 1656.21KB - stringify: 12.85ms, write: 12.20ms, total: 25.04ms
[ControllerStorage PERF] setItem TokenListController - 2137.56KB - stringify: 12.47ms, write: 11.40ms, total: 23.87ms
[ControllerStorage PERF] setItem TokenListController - 4068.75KB - stringify: 22.00ms, write: 22.62ms, total: 44.62ms

Main Branch - Add New Chain (Monad)

[ControllerStorage PERF] setItem TokenListController - 4102.55KB - stringify: 23.52ms, write: 21.82ms, total: 45.34ms

🔧 Performance Logging Code (Main Branch)

The following code was added to app/store/persistConfig/index.ts to capture performance metrics:

Read Performance Logging (getAllPersistedState)

async getAllPersistedState(): Promise<Record<string, unknown>> {
  // eslint-disable-next-line no-console
  console.warn('[ControllerStorage PERF] getAllPersistedState started');
  const totalStart = performance.now();
  try {
    const backgroundState: Record<string, unknown> = {};

    await Promise.all(
      Array.from(
        new Set(
          Array.from(BACKGROUND_STATE_CHANGE_EVENT_NAMES).map(
            (eventName) => eventName.split(':')[0],
          ),
        ),
      ).map(async (controllerName) => {
        const key = `persist:${controllerName}`;
        const startTime = performance.now();
        try {
          const data = await FilesystemStorage.getItem(key);
          if (data) {
            const parseStart = performance.now();
            const parsedData = JSON.parse(data);
            const parseDuration = performance.now() - parseStart;
            const totalDuration = performance.now() - startTime;

            // Log performance for TokenListController specifically
            if (controllerName === 'TokenListController') {
              const sizeKB = (data.length / 1024).toFixed(2);
              // eslint-disable-next-line no-console
              console.warn(
                `[ControllerStorage PERF] ${controllerName} - ${sizeKB}KB - ` +
                  `read: ${(totalDuration - parseDuration).toFixed(2)}ms, ` +
                  `parse: ${parseDuration.toFixed(2)}ms, ` +
                  `total: ${totalDuration.toFixed(2)}ms`,
              );
            }
            // ... rest of the function
          }
        } catch (error) {
          // error handling
        }
      }),
    );

    const totalDuration = performance.now() - totalStart;
    // eslint-disable-next-line no-console
    console.warn(
      `[ControllerStorage PERF] getAllPersistedState complete - ${totalDuration.toFixed(2)}ms`,
    );

    return { backgroundState };
  } catch (error) {
    // error handling
  }
}

Write Performance Logging (createPersistController)

export const createPersistController = (debounceMs: number = 200) =>
  debounce(async (filteredState: unknown, controllerName: string) => {
    const startTime = performance.now();
    try {
      const stringifyStart = performance.now();
      const serialized = JSON.stringify(filteredState);
      const stringifyDuration = performance.now() - stringifyStart;

      await ControllerStorage.setItem(`persist:${controllerName}`, serialized);

      const totalDuration = performance.now() - startTime;
      if (controllerName === 'TokenListController') {
        const sizeKB = (serialized.length / 1024).toFixed(2);
        // eslint-disable-next-line no-console
        console.warn(
          `[ControllerStorage PERF] setItem ${controllerName} - ${sizeKB}KB - ` +
            `stringify: ${stringifyDuration.toFixed(2)}ms, ` +
            `write: ${(totalDuration - stringifyDuration).toFixed(2)}ms, ` +
            `total: ${totalDuration.toFixed(2)}ms`,
        );
      }
      Logger.log(`${controllerName} state persisted successfully`);
    } catch (error) {
      // error handling
    }
  }, debounceMs);

🔧 Performance Logging Code (This PR)

The following code was added to app/core/Engine/controllers/storage-service-init.ts to capture performance metrics for the per-chain storage:

getItem - Read Performance Logging

async getItem(namespace: string, key: string): Promise<StorageGetResult> {
  // eslint-disable-next-line no-console
  console.warn(`[StorageService DEBUG] getItem called: ${namespace}:${key}`);
  const startTime = performance.now();
  try {
    const fullKey = `${STORAGE_KEY_PREFIX}${namespace}:${key}`;
    const serialized = await FilesystemStorage.getItem(fullKey);

    // Key not found - return empty object
    if (serialized === undefined || serialized === null) {
      const duration = performance.now() - startTime;
      if (
        key.includes('token') ||
        key.includes('Token') ||
        namespace.includes('Token')
      ) {
        // eslint-disable-next-line no-console
        console.warn(
          `[StorageService PERF] getItem ${namespace}:${key} - NOT FOUND - ${duration.toFixed(2)}ms`,
        );
      }
      return {};
    }

    const parseStart = performance.now();
    const result = JSON.parse(serialized) as Json;
    const parseDuration = performance.now() - parseStart;
    const totalDuration = performance.now() - startTime;

    if (
      key.includes('token') ||
      key.includes('Token') ||
      namespace.includes('Token')
    ) {
      const sizeKB = (serialized.length / 1024).toFixed(2);
      // eslint-disable-next-line no-console
      console.warn(
        `[StorageService PERF] getItem ${namespace}:${key} - ${sizeKB}KB - ` +
          `read: ${(totalDuration - parseDuration).toFixed(2)}ms, ` +
          `parse: ${parseDuration.toFixed(2)}ms, ` +
          `total: ${totalDuration.toFixed(2)}ms`,
      );
    }

    return { result };
  } catch (error) {
    // error handling
  }
}

setItem - Write Performance Logging

async setItem(namespace: string, key: string, value: Json): Promise<void> {
  // eslint-disable-next-line no-console
  console.warn(`[StorageService DEBUG] setItem called: ${namespace}:${key}`);
  const startTime = performance.now();
  try {
    const fullKey = `${STORAGE_KEY_PREFIX}${namespace}:${key}`;

    const stringifyStart = performance.now();
    const serialized = JSON.stringify(value);
    const stringifyDuration = performance.now() - stringifyStart;

    await FilesystemStorage.setItem(fullKey, serialized, Device.isIos());

    const totalDuration = performance.now() - startTime;

    if (
      key.includes('token') ||
      key.includes('Token') ||
      namespace.includes('Token')
    ) {
      const sizeKB = (serialized.length / 1024).toFixed(2);
      // eslint-disable-next-line no-console
      console.warn(
        `[StorageService PERF] setItem ${namespace}:${key} - ${sizeKB}KB - ` +
          `stringify: ${stringifyDuration.toFixed(2)}ms, ` +
          `write: ${(totalDuration - stringifyDuration).toFixed(2)}ms, ` +
          `total: ${totalDuration.toFixed(2)}ms`,
      );
    }
  } catch (error) {
    // error handling
  }
}

getAllKeys - Key Enumeration Logging

async getAllKeys(namespace: string): Promise<string[]> {
  // eslint-disable-next-line no-console
  console.warn(`[StorageService DEBUG] getAllKeys called: ${namespace}`);
  const startTime = performance.now();
  try {
    const allKeys = await FilesystemStorage.getAllKeys();

    if (!allKeys) {
      const duration = performance.now() - startTime;
      if (namespace.includes('Token')) {
        // eslint-disable-next-line no-console
        console.warn(
          `[StorageService PERF] getAllKeys ${namespace} - 0 keys - ${duration.toFixed(2)}ms`,
        );
      }
      return [];
    }

    const prefix = `${STORAGE_KEY_PREFIX}${namespace}:`;
    const filteredKeys = allKeys
      .filter((key) => key.startsWith(prefix))
      .map((key) => key.slice(prefix.length));

    const duration = performance.now() - startTime;
    if (namespace.includes('Token')) {
      // eslint-disable-next-line no-console
      console.warn(
        `[StorageService PERF] getAllKeys ${namespace} - ${filteredKeys.length} keys found - ${duration.toFixed(2)}ms`,
      );
    }

    return filteredKeys;
  } catch (error) {
    // error handling
  }
}

Changelog

CHANGELOG entry: integrates per chain file save for tokenListController.

Related issues

Related: MetaMask/core#7413

Manual testing steps

Feature: my feature name

  Scenario: user [verb for user action]
    Given [describe expected initial app state]

    When user [verb for user action]
    Then [describe expected outcome]

Screenshots/Recordings

Before

After

Pre-merge author checklist

I’ve followed MetaMask Contributor Docs and MetaMask Mobile Coding Standards.
I've completed the PR template to the best of my ability
I’ve included tests if applicable
I’ve documented my code using JSDoc format if applicable
I’ve applied the right labels on the PR (see labeling guidelines). Not required for external contributors.

Pre-merge reviewer checklist

I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed).
I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots.

Note

Switches TokenListController to per-chain StorageService-backed caching with init migration, per-chain saves, and messenger wiring.

TokenListController (assets-controllers patch)
- Set tokensChainsCache.persist to false; introduce per-chain storage keys tokensChainsCache:<chainId>.
- Add helpers: _loadCacheFromStorage, _saveChainCacheToStorage, _migrateStateToStorage, _getChainStorageKey, _stopPolling.
- On fetch: update state with { data, timestamp } and persist only the affected chain; handle empty responses similarly.
- On clear: wipe in-memory cache and remove all per-chain items (and legacy single-file) from StorageService.
- Load cache on init and migrate legacy persisted state if needed; use private fields for mutex/interval/abort/chainId.
- Type changes: export DataCache; include StorageService actions in messenger types; make clearingTokenListData async.
App integration
- Pass persistedState.TokenListController to controller init.
- Delegate StorageService:getAllKeys|getItem|setItem|removeItem actions in token-list-controller messenger.
Dependencies
- Add @metamask/storage-service and apply patch to @metamask/assets-controllers; update yarn.lock.

^{Written by Cursor Bugbot for commit 070448e. This will update automatically on new commits. Configure here.}

github-actions · 2025-12-15T14:35:51Z

CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes.

sahar-fehri · 2025-12-15T14:56:21Z

.yarn/patches/@metamask-assets-controllers-npm-94.1.0-481519c877.patch

+    if (typeof state === "function" ? receiver !== state || !f : !state.has(receiver)) throw new TypeError("Cannot write private member to an object whose class did not declare it");
+    return (kind === "a" ? f.call(receiver, value) : f ? f.value = value : state.set(receiver, value)), value;
+};
+var _TokenListController_instances, _a, _TokenListController_mutex, _TokenListController_storageKeyPrefix, _TokenListController_getChainStorageKey, _TokenListController_intervalId, _TokenListController_intervalDelay, _TokenListController_cacheRefreshThreshold, _TokenListController_chainId, _TokenListController_abortController, _TokenListController_loadCacheFromStorage, _TokenListController_saveChainCacheToStorage, _TokenListController_migrateStateToStorage, _TokenListController_onNetworkControllerStateChange, _TokenListController_stopPolling, _TokenListController_startDeprecatedPolling;


patch file to be removed after release, this is just for testing

sahar-fehri · 2025-12-15T18:51:18Z

app/core/Engine/controllers/storage-service-init.ts

+  async getAllKeys(namespace: string): Promise<string[]> {
+    // eslint-disable-next-line no-console
+    console.warn(`[StorageService DEBUG] getAllKeys called: ${namespace}`);
+    const startTime = performance.now();


perf logs to be cleaned up

sahar-fehri · 2025-12-15T18:52:03Z

app/store/persistConfig/index.ts

+              const totalDuration = performance.now() - startTime;
+
+              // Log performance for TokenListController specifically
+              if (controllerName === 'TokenListController') {


Change to be reverted; only for testing

sahar-fehri · 2025-12-15T21:26:47Z

app/core/Engine/controllers/storage-service-init.ts

-      return allKeys
+      const filteredKeys = allKeys
        .filter((key) => key.startsWith(prefix))
        .map((key) => key.slice(prefix.length));


Filter to only keys that belong to the requested namespace

.yarn/patches/@metamask-assets-controllers-npm-94.0.0-0b6df6cf06.patch

github-actions · 2025-12-16T09:02:45Z

🔍 Smart E2E Test Selection

Selected E2E tags: SmokeCore, SmokeAssets, SmokeWalletPlatform, SmokeNetworkExpansion, SmokeMultiChainAPI
Risk Level: high
AI Confidence: 85%

click to see 🤖 AI reasoning details

This PR introduces significant changes to the TokenListController, a core component in the Engine that manages token list caching across multiple chains.

Key changes:

Storage mechanism change: The TokenListController's tokensChainsCache is being migrated from framework-managed persistence (persist: true) to a new StorageService-based per-chain file storage. This is a fundamental change to how token data is persisted.
New StorageService integration: The messenger now delegates StorageService actions (getAllKeys, setItem, getItem, removeItem) to allow the controller to manage its own persistence.
Migration logic: The patch includes migration code to handle users upgrading from the old state-based persistence to the new per-chain storage files.
Async changes: clearingTokenListData() is now async to handle StorageService cleanup.
Persisted state initialization: The controller init now passes persistedState.TokenListController to restore state.

These changes are HIGH RISK because:

They modify core Engine controller initialization
They change the persistence mechanism for token data
They affect multi-chain token caching behavior
They involve migration logic that could affect existing users
TokenListController is used by token detection, asset display, and network switching

Selected tags rationale:

SmokeCore: Core wallet functionality and Engine changes
SmokeAssets: Token list management directly affects asset display and token detection
SmokeWalletPlatform: Core wallet operations depend on token lists
SmokeNetworkExpansion: Per-chain caching affects network configuration and multi-chain support
SmokeMultiChainAPI: Multi-chain token caching is being modified

View GitHub Actions results

cursor · 2025-12-16T09:06:03Z

.yarn/patches/@metamask-assets-controllers-npm-94.1.0-481519c877.patch

+        if (Object.keys(loadedCache).length > 0) {
+            this.update((state) => {
+                state.tokensChainsCache = loadedCache;
+            });


Bug: Async cache load overwrites concurrent fetch data

The #loadCacheFromStorage method runs asynchronously in the constructor without using the mutex that protects fetchTokenList. When it completes, it performs a complete replacement of tokensChainsCache (state.tokensChainsCache = loadedCache) rather than merging. If fetchTokenList runs and updates state while the async load is in progress, those updates are overwritten when #loadCacheFromStorage completes with stale data. This can cause freshly fetched token data to be lost from state, potentially requiring redundant API calls or causing temporary UI inconsistencies.

Additional Locations (1)

.yarn/patches/@metamask-assets-controllers-npm-94.1.0-481519c877.patch#L49-L58

sonarqubecloud · 2025-12-16T09:24:30Z

Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

metamaskbot added the team-assets label Dec 15, 2025

github-actions bot added the size-XL label Dec 15, 2025

sahar-fehri commented Dec 15, 2025

View reviewed changes

sahar-fehri changed the title ~~Feat/integrate token list controller storage service~~ feat: integrate token list controller storage service Dec 15, 2025

sahar-fehri changed the base branch from main to feature/storage-service December 15, 2025 21:22

sahar-fehri commented Dec 15, 2025

View reviewed changes

sahar-fehri marked this pull request as ready for review December 15, 2025 21:29

sahar-fehri requested a review from a team as a code owner December 15, 2025 21:29

sahar-fehri added the DO-NOT-MERGE Pull requests that should not be merged label Dec 15, 2025

Base automatically changed from feature/storage-service to main December 16, 2025 01:26

sahar-fehri requested a review from a team as a code owner December 16, 2025 08:23

github-actions bot added size-L and removed size-XL labels Dec 16, 2025

cursor bot reviewed Dec 16, 2025

View reviewed changes

.yarn/patches/@metamask-assets-controllers-npm-94.0.0-0b6df6cf06.patch Outdated Show resolved Hide resolved

chore: integrate tokenListController with StorageService

070448e

sahar-fehri force-pushed the feat/integrate-tokenListController-storageService branch from dfa8049 to 070448e Compare December 16, 2025 08:59

Prithpal-Sooriya approved these changes Dec 16, 2025

View reviewed changes

cursor bot reviewed Dec 16, 2025

View reviewed changes

georgewrmarshall removed the request for review from a team December 18, 2025 02:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

feat: integrate token list controller storage service #24019

feat: integrate token list controller storage service #24019

sahar-fehri commented Dec 15, 2025 •

edited by cursor bot

Loading

Uh oh!

github-actions bot commented Dec 15, 2025

Uh oh!

sahar-fehri Dec 15, 2025

Uh oh!

sahar-fehri Dec 15, 2025

Uh oh!

sahar-fehri Dec 15, 2025

Uh oh!

sahar-fehri Dec 15, 2025

Uh oh!

Uh oh!

github-actions bot commented Dec 16, 2025

Uh oh!

cursor bot Dec 16, 2025

Uh oh!

sonarqubecloud bot commented Dec 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

feat: integrate token list controller storage service #24019

Are you sure you want to change the base?

feat: integrate token list controller storage service #24019

Conversation

sahar-fehri commented Dec 15, 2025 • edited by cursor bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Performance Comparison: Per-Chain Token Cache Storage

📊 Complete Performance Comparison

Cold Restart

Onboarding

Add New Chain (Monad)

Summary

📋 Captured Logs

This PR - Cold Restart

This PR - Onboarding

This PR - Add New Chain (Monad)

Main Branch - Cold Restart

Main Branch - Onboarding

Main Branch - Add New Chain (Monad)

🔧 Performance Logging Code (Main Branch)

Read Performance Logging (getAllPersistedState)

Write Performance Logging (createPersistController)

🔧 Performance Logging Code (This PR)

getItem - Read Performance Logging

setItem - Write Performance Logging

getAllKeys - Key Enumeration Logging

Changelog

Related issues

Manual testing steps

Screenshots/Recordings

Before

After

Pre-merge author checklist

Pre-merge reviewer checklist

Uh oh!

github-actions bot commented Dec 15, 2025

Uh oh!

sahar-fehri Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

sahar-fehri Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

sahar-fehri Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

sahar-fehri Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Dec 16, 2025

🔍 Smart E2E Test Selection

Uh oh!

cursor bot Dec 16, 2025

Choose a reason for hiding this comment

Bug: Async cache load overwrites concurrent fetch data

Uh oh!

sonarqubecloud bot commented Dec 16, 2025

Quality Gate passed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sahar-fehri commented Dec 15, 2025 •

edited by cursor bot

Loading