Steam Inventory Value prices a Steam inventory from a cache of market prices. A crawler inside Postgres keeps that cache fresh: every minute it may fetch two pages of a game's market catalogue, 100 items a page. The budget is small on purpose, because Steam rations requests.
Two pages a minute has to be shared between every game we support. This post is about how we shared it, why that quietly failed for new games, and the small rule that fixed it.
The rotation
Each game has a refresh interval: 6 hours for Dota 2 and CS2, 12 for TF2 and Rust, 24 for Steam trading cards. The crawler picks the game that is most overdue relative to its own interval:
select appid from crawl_apps
where enabled
order by (cycle_started_at is null) desc,
extract(epoch from now() - cycle_started_at) / (refresh_hours * 3600) desc nulls first,
sort
limit 1
A game that started its round 12 hours ago with a 6-hour interval scores 2.0. One that started 12 hours ago with a 24-hour interval scores 0.5. Highest score goes next. When a game finishes a round it starts the next one, its clock resets, and it drops to the back.
This is a weighted round-robin, and for the games already in it, it is fair: popular games get more turns, and nobody is skipped forever.
The numbers we didn't add up
The catalogues are not the same size:
| Game | Items | Pages |
|---|---|---|
| Steam trading cards | about 325,000 | 3,254 |
| Team Fortress 2 | 44,617 | 447 |
| Dota 2 | 34,568 | 346 |
| CS2 | 13,352 | 134 |
| Rust | 5,578 | 56 |
That is about 4,200 pages. At two pages a minute, one pass over everything takes about 35 hours.
So no game is ever refreshed at its interval. In a steady state every game gets its turn when its score reaches roughly the same value, and that value is the total work divided by the budget. For us it comes to about 2.2. A "6-hour" game is refreshed about every 13 hours, and a "24-hour" game about every 53.
We could live with that: a cached price stays valid for 7 days, and items a customer is actually waiting for are fetched one by one, ahead of any catalogue page.
Where it broke
On 4 November we added PUBG, 357 items, four pages of work.
A new game has never been crawled, so it goes first, once. It fetched its first page, its clock started, and its score became zero. A day later it was still at 100 of 357 items.
Nothing was wrong with the code. PUBG was simply waiting for its score to climb past games that had been overdue for a day and a half. With a 12-hour interval that takes about 26 hours. The three games we added the next day had 24-hour intervals: they would have waited about 53 hours for a few minutes of crawling.
The "crawl again now" button in our admin panel had the same flaw. It cleared the clock, the game got one page, and then it was at the back again.
The fix
A rotation that is fair in the steady state says nothing about how a newcomer gets into it. So we added one rule: a game that has never finished a round goes first, smallest catalogue first. After its first full round it joins the rotation like everyone else.
One column counts finished rounds:
alter table crawl_apps add column if not exists cycles_done int not null default 0;
The function that advances a round adds one when the round completes, and the picker puts the games at zero in front:
select appid from (
select a.*,
(a.cycles_done = 0 and (
select count(*) from crawl_log l
where l.appid = a.appid and l.kind = 'page'
and l.status is distinct from 200 and l.at > now() - interval '1 hour'
) < 3) as catching_up
from crawl_apps a
where a.enabled
) a
order by catching_up desc,
case when catching_up then coalesce(total, 0) end asc nulls last,
(cycle_started_at is null) desc,
extract(epoch from now() - cycle_started_at) / (refresh_hours * 3600) desc nulls first,
sort
limit 1
The inner condition is a guard. A game that jumps the queue and then fails on every request would block every other catalogue. So a newcomer whose page requests failed three times in the last hour loses its priority until they stop failing.
Smallest first is the classic shortest-job-first idea: the small catalogues finish in minutes, so the fewest games are left waiting.
Checking before touching production
The crawler's SQL has tests that run the real migrations in an in-process Postgres (PGlite). We added three: a new game finishes its first round before the rotation resumes, the smaller of two newcomers goes first, and a failing newcomer doesn't jump the queue.
Then we ran the new ordering as a plain read-only query against production to see what it would pick. It listed the four waiting games, smallest first, then the old rotation unchanged.
The result
PUBG finished two minutes after the change went in. The next game, with 460 items, finished two minutes after that. While it ran we queued three more games. All seven newcomers, about 17,700 items between them, were done an hour and a half after the change.
The cost was that the big catalogues paused for that time. With a 7-day price validity, nobody noticed.
What we took from it
- A scheduler that is fair between its members still needs a rule for joining.
- Add up the work. Our intervals promised 6, 12 and 24 hours; the budget could deliver about 2.2 times that. The intervals were really weights.
- Give anything that can jump a queue a way to lose that privilege.
Try it
Steam Inventory Value is free and needs no sign-in: https://steam.2cstudio.app. Paste a Steam profile link and it prices the whole inventory.
Written by 2C Studio, a small studio.