Benchmarketing 101
Today’s issue:
- Impolite data structures
- Lukewarm Mr. Pibb
- The crypto trade of a lifetime
Welcome to #142 - you can read it online here.
The Main Thing
When the hot new OSS tool is not quite as blazing as it claimed
Benchmarketing 101
FOMO is a hell of a drug. That’s why, no matter how rational we *think* we are, our lizard brains simply can’t resist a “10x lightning fast” benchmark for a new OSS tool.
OSS creators know this, which is why some of them spend a lot of time crafting benchmarks that always show their tool as being #1 Most Blazingest. And while it’s natural to want to highlight the strengths of your own project, sometimes this benchmarketing (🥁) can cause some real confusion.
Vercel recently came under fire for their Turbopack-vs-Vite benchmarks, Bun was questioned (but then mostly defended for their SSR benchmarks, and the list goes on. So, as concerned citizens of JavaScript-land, we wanted to share a few benchmarking practices to look out for when evaluating various OSS claims.
Let’s start with a few red flags 🚩:
-
Not public: If you can’t run a benchmark yourself, you should probably be skeptical.
-
Lacks context: Does the benchmark include context about what their assumptions are? Measuring performance on “hello world” can be wildly different than a huge application.
-
Unclear methodology: Things like what machine the benchmarks are run on, how many times they ran, and what use cases are covered, all matter. If those things are unclear, your spidey senses should be going off.
Here are a few best practices we love to see 🤩:
-
Everyone agrees on what to measure: Standardized benchmarks ftw. Codefied standards are more common for larger and more established projects like browsers — but the more we can all agree on, the better.
-
You benchmark against yourself: This is the best way to give comparisons that are truly 🍎’s to 🍎’s, and there’s a lot to be said for transparently running your own race.
Bottom Line: In a perfect world, we’d have some sort of third-party benchmark for benchmarking the quality of every benchmark. But since that job sounds like actual hell, we’ll probably just have to keep trying to decipher the data ourselves — and celebrate the times when great benchmarking helps us all win.
    |
Our Friends
(With Benefits)
Can't wait to hand-roll those back-end APIs
Directus can help you
You know what’s cool? Building things your users love. You know what isn’t cool? Building backend APIs so that butthead from marketing can pretend he’s a data scientist.
That’s why we love Directus — it sits on top of your SQL database and comes with a bunch of features, one of them being that it dynamically generates REST and GraphQL APIs for you and your team.
Even better, it also spins up a no-code app so that anyone (even Mr. Butthead) can easily access company data, build internal dashboards, make data visualizations, and stop pinging you on Slack every 20 minutes to ask for help 🥹.
Directus is a legit open-source product (18k GitHub stars, 20m downloads) that’s completely free to self host, and their new cloud platform looks really cool too: global CDN, end-to-end project provisioning, and lots more.
Try it for free, and never get lost in the Forbidden Forest your company’s SQL database again.
The Job Board
 | Lead Technical Architect | ||
| |||
The UK Home Office is looking for an experienced engineer to help define, own, and contribute to its technical roadmap and strategy as they build applications and services used by the UK government and its citizens. | |||
Powered By: |
 | Cloud Architect | |||
| ||||
Capco is the largest financial services consulting firm in the world, and they're looking for an experienced cloud engineer to lead new projects for their clients, which ranges from FinTech companies to global banks. | ||||
Powered By: |
Pop Quiz
Sponsored by JSworld Conference 2023
The #1 JS conference in the world is back in Amsterdam on Feb. 8-10, with 45 speakers from the Vite Core Team, Remix & Angular teams, and lots more.
What gets logged?
for (var i = 0; i < 5; i++) {
setTimeout(() => console.log(i), 1000)
}
Cool Bits
-
FusionAuth created this comprehensive ebook on Breaking Down JSON Web Tokens. Now I just need them to write an ebook that helps me figure out the exchange rate between all of my FTT Tokens and Chuck E. Cheese Tokens, so I can complete the trade of a lifetime. [sponsored]
-
Ruben Casas wrote an article called Is React going anywhere? This marks the 6th year in a row we’ve seen an article like this written.
-
Hey GPT-3: write me a Taylor Swift song about a chunky, cross-platform JS framework getting some new Chromium upgrades. […processing… processing…] “I don’t know about you, but I’m feeling Electron 22.”
-
Dominik (React Query maintainer) wrote Inside React Query which is an interesting walk through of how React Query works under the hood.
-
Guido van Rossum (creator of Python) and Lex Fridman (smart podcast guy) took some time to talk about TypeScript vs JavaScript. It’s not terribly insightful for those of us in the weeds, but it’s always interesting to see our small part of the internet get discussed.
-
Johannes Kettmann wrote about Data Structures In Frontend JavaScript In The Real World. Now we’ll finally know what happens when data structures stop being polite, and start getting real.
-
Sophie wrote about how and why her team migrated from Vue 2 to Svelte, instead of upgrading to Vue 3. It reminded me of when I tried migrating to Azerbaijan instead of finishing my 9th grade social studies paper.
-
The TC39 Committee just met, and everyone’s very excited about this Stage-3 proposal that brings some helpful new methods to
set.prototype. What kind of snacks do you think they have at these TC39 meetings? Do they keep it classy with pears and fondue? Or is there just a giant communal bowl of Doritos and a two-liter bottle of lukewarm Mr. Pibb?
What gets logged?
for (var i = 0; i < 5; i++) {
setTimeout(() => console.log(i), 1000)
}
A little throw back for all you old heads. This used to be everyone’s favorite JavaScript interview question.
This code will log 5 5 times.
The reason for this is because we declared our variable with var, which is function scoped. By the time our functions given to setTimeout run, i will already be 5 and every function will reference that same variable. To fix this, use let which is block scoped and will give each iteration of our for loop its own i.
for (let i = 0; i < 5; i++) {
setTimeout(() => console.log(i), 1000)
}