How I built 1k Checkboxes using websockets

The Idea

The idea to make this site was from 1 million checkboxes by itseieio. My first plan was to create a similar site and scale it to its level, but later I changed my mind to first work on a basic MVP and then later on move towards the 1 million checkbox goal. So here is my journey towards the 1k checkbox.

The Stack

1. Node.js

2. Express

3. Socket.IO

4. Redis ( Valkey )

5. Iris ( my own OIDC server )

Building The Backend

1. For this project the first challenge for me was to use websockets for real time sync across multiple users. I used socket.io for the web sockets part, all it does is emit and listen to events whenever the state of the checkbox was changed.

   socket.on("client:checkbox:changed", async (data) => {}
   

2. Now, during a live session if a new client connects, it doesn't know which checkbox was checked or unchecked. To solve this problem I had two ways:
   a. Use local memory to store states
   b. Use redis to store states

   I used Redis pub/sub to store the states of the checkboxes in an array containing boolean values, True meant the checkbox was checked and vice versa.

   I created 3 connections:
   1. Publisher: This connection broadcasts the messages to the redis channel.
   2. Subscriber: This connection is dedicated to listening the messages broadcasted by the publisher.
   3. "Standard" redis: This connection is for data setting and retrieving.

3. Now the next challenge was the abuse of server. Clients had full freedom to spam the checkboxes, hence abusing the servers and to fix that I first simply used my custom rate-limiting by creating a HashMap and storing the last operation time of the client with the socket id as its key and hence not letting the client interact with the checkboxes while the user is still on cooldown.

   NOTE: This had a problem which I fixed after working on auth

The Auth Nightmare

The toughest challenge was to implement authentication (oAuth) using my OIDC server i.e Iris, oh man this was the first time for me and It was a nightmare. I had already built Iris (will be sharing its journey too).

1. While trying to cache the public key from the jwks endpoint I realised I had not used the standard way for an OIDC server. The public keys should be provided in an array but I was straight away returning the keys as an object.
   I used jwks-rsa library for caching the public key btw!

   // How it expected the response:
   
   {
     "keys": [
       {
         "e": "AQAB",
         "kty": "RSA",
         "n": "0WjG8_smfA2mgPMnvSPDbLjSCX7z_TQEMVZq2uuG4FIgXq92walz9ATdNP5oLvcp1WVxXaInmL2UvHOPuH0azaDGn3Ta9qz8R1AowAqwbsC_VM_-ofeEt7nOUe4EHvfVbd71fXTJ8DVhlRxKQBfEy2kGnILpL3Gk2_HDRC0D1S1oNjBu0-SsS4PNL3NZUwGp_WmUdop6qz8JgxS1bct6w2Di_UywDX70rFbm1Si0CXlCxBUKfIrSzI0_AyT4s2S_OvzjBrRFaTrdFSBNJ84nY3hxj4DKALaEXWScH5B94w-kGpsAvY6c8_JDIXtHguv6nrXT8sfcWnNLf7PU0EK71yQ",
         "kid": "081afe27305fce5a",
         "use": "sig",
         "alg": "RS256"
       }
     ]
   }
   
   // How I was sending the response:
   
   {
       "e": "AQAB",
       "kty": "RSA",
       "n": "0WjG8_smfA2mgPMnvSPDbLjSCX7z_TQEMVZq2uuG4FIgXq92walz9ATdNP5oLvcp1WVxXaInmL2UvHOPuH0azaDGn3Ta9qz8R1AowAqwbsC_VM_-ofeEt7nOUe4EHvfVbd71fXTJ8DVhlRxKQBfEy2kGnILpL3Gk2_HDRC0D1S1oNjBu0-SsS4PNL3NZUwGp_WmUdop6qz8JgxS1bct6w2Di_UywDX70rFbm1Si0CXlCxBUKfIrSzI0_AyT4s2S_OvzjBrRFaTrdFSBNJ84nY3hxj4DKALaEXWScH5B94w-kGpsAvY6c8_JDIXtHguv6nrXT8sfcWnNLf7PU0EK71yQ",
       "kid": "081afe27305fce5a",
       "use": "sig",
       "alg": "RS256"
   }
   

2. I was first stuck in solving the issue on how to send the access token to the frontend safely and set it to local storage but I could not find any solution, so
   After talking with my mentor and researching furthur, I realised httpOnly cookies are actually the more secure choice over localStorage, JavaScript can't access them, making them immune to XSS attacks. What felt like a compromise was actually the right call.

   

3. After finally getting the tokens as cookies, turns out sameSite property was set to strict and It was breaking the callback flow. I got to know that if the property is set to strict, it prevents sending cookies on cross-site requests, which I was doing because OIDC server is deployed separately. So I set the sameSite property to lax. Which is opposite of strict and a standard practice in oAuth.

4. While implementing the refresh tokens, I got to know I was not rotating the tokens correctly, I was storing the old refresh token back into the db instead of storing the new one hence breaking the whole flow.

Frontend Flow

1. The requireAuth -> tryRefresh -> reload Pattern: This is the standard guard for the site, which verifies whether the client has a access token and then lets it to access to the main page.

   - requireAuth: This function checks if the user has a valid access token in memory or via a cookie. If missing, it doesn't immediately give up; it calls tryRefresh.

   - tryRefresh: This attempts to hit the /auth/refresh-token endpoint. If the server validates the refresh token, it issues a new access token cookie.

   - The Reload/Retry: Once refreshed, the page reloads completely so all subsequent requests use the new cookies cleanly rather than retrying in the same cycle which caused race conditions. If tryRefresh fails, the user is redirected to /login.

2. Singleton tryRefresh: Preventing Parallel Calls
   This was a critical performance and security optimization. When the main page loads, multiple components might realize the token is expired at the exact same time.
   I faced the similar issue all the components called refresh token endpoint simultaneously for new tokens. In which I noticed a pattern.

   - The first request succeeded
   - The rest of the requests were using the old token and causing the client to completely get logged out hence redirecting it to the login page

   Solution: I stored the "refreshing" state in a variable. If a refresh is already in progress, other callers wait for the same promise instead of starting a new one.

   btw, I never thought of this way, so I had to use AI for this solution :D

   let refreshPromise = null;
   
   async function tryRefresh() {
     if (refreshPromise) return refreshPromise; // Return the existing process
   
     refreshPromise = (async () => {
       try {
         const res = await fetch("/auth/refresh-token", { method: "POST" });
         return res.ok;
       } finally {
         refreshPromise = null; // Reset once finished
       }
     })();
   
     return refreshPromise;
   }
   

Fixing Rate Limiting

The original rate limiting used socket.id as the key. The problem was that every page refresh creates a new socket with a new ID, hence resetting the cooldown. The fix was to verify the user's access token on socket connection and use user.sub (i.e user id from JWT) as the key instead. Instead of keeping the rate limiting in memory Hash map, I shifted it to Redis with TTL (time to live), so it works across multiple server instances and auto-expires without any cleanup code

Production 😭

The thing that took so much of my time after auth was this. It was all about:

• Fixing cors errors

• Deploying Redis Server and I got to know that it has 2 different urls one is internal url and one is external. I was trying so hard to create a connection using the internal one. After a single question to AI I got to know that I have to use the external url which starts with rediss:// instead of redis://

• The biggest mistake was to trying to host my OIDC on vercel. Literally had to deploy it again on render and It fixed so many of my problems.

• Also please try to use environment variables even while coding because, I had hardcoded the localhost url of the OIDC server in JwksClient. I kept wondering why it was working locally and not on production (skill issue ig) 😭

• One solution that I am yet to get is that I have to use the keys as env variables because somehow the deployed ones are not able to read the .pem files even though the path is correct.
  Here is how I am doing it for now 😓

  const formatPEM = (key: string, type: string) => {
    if (!key) return "";
  
    // 1. Remove any existing headers/footers and whitespace
    const cleanKey = key
      .replace(/-----BEGIN (.*)-----/, "")
      .replace(/-----END (.*)-----/, "")
      .replace(/\s/g, "");
  
    // 2. Re-wrap the key at 64 characters (Standard PEM requirement)
  
    const wrappedKey = (cleanKey.match(/.{1,64}/g) ?? []).join("\n");
  
    // 3. Add the correct headers back
    return `-----BEGIN \({type}-----\n\){wrappedKey}\n-----END ${type}-----`;
  };
  
  export const PUBLIC_KEY = formatPEM(process.env.PUBLIC_KEY!, "PUBLIC KEY");
  export const PRIVATE_KEY = formatPEM(process.env.PRIVATE_KEY!, "PRIVATE KEY");
  

Thank You

Thanks for reading so far, I hope you learnt something from this blog. This was my first time documenting a whole journey of a project in short.
Consider following me on X ( am very active there :D ) -> @wedan_
Github Repo: checkboxes