‘Working in public’ is a phrase I hear often and it’s synonymous with the ‘builder.’ But I like having a blog because it allows me to think in public. I think it’s an investment of compounding gains that takes very little investment. I implore everyone to start one!
This year I didn’t write as much on mine. I’ve never had it as a strict obligation but although I think journalling has a similar effect, I did find that the growing optionality of writing here a factor in ‘thinking less.’
Consolidation is very powerful. In some ways, it’s the preservation of trains of thought as relics of fascination. It’s the sharpening of mental motions into clarity. A kind of clarity that is the melding of intuition, reflection, and personal narrative-making: an introspective philosophy. I think the kind of clarity that is the basis of action comes from another form of writing that I don’t do here.
While I did write a fair amount this year through other outlets, the audience of all those writings weren’t me. I think this makes a big difference in how much they benefited my own mind as a result. For that reason, I think having a blog can and should be a selfish act for most. I think having a Substack where I write spirited essays about my subjects of study would be very fun, but it wouldn’t be a blog in the conception of one that I have in mind. That said, starting one would be interesting.
It was in spring 2022 when a friend and I spoke about starting our own. He had already written a few and I had none. I thought to myself why would I start my own? No one would read it. I did anyway. First, you make the front page and you’re tasked with coming up with a little blurb about yourself. What do you do? That’s a good question to ask. I remember this used to be a list of fleeting topics of interest. Now, it’s more a list of places I’ve worked. Neither is better than the other I think. I’m still working on my introductions.
Fun fact: I’ve had a website for a while, but the other one was made by my dad. I’d write bad essays about the political and economic state of the world on it and then send it to him to upload it. I’d later learn basic web development and figure out Github pages which marked my digital sovereignty.
I got my first official internship at a small local magazine with it. The founder told me that it was the main reason I was hired. I was also a terrible employee. At the time, Ramadan meant staying up until the crack of dawn which meant that I was half awake for the calls we had.
Anyway, the blog was never about other people and yet it’s paid major dividends. It’s most definitely not a design portfolio but I find the weird, kinda ugly, Y2K aesthetic kind of charming. It feels like an ode to my young self that this exists so I hope it continues to.
]]>One great thing about the internet is that advice is ample. While I think directly talking to someone you’ve reached out to is valuable, there is a lot to gain from seeking out expert opinion in the wild. This year, I spent a lot of time gaining exposure to research. Research encompasses a broad spectrum of activities, but in my infancy pursuing it, I would define it as the pursuit of understanding as a tool. And I think this pursuit is very cool.
Journalists seek to understand, and their goal is to communicate that understanding to the public. The engineer seeks to understand, so that they can tinker, fix, and build—often for the public. The (academic) researcher seeks to understand, so that they can build new tools of thought that others can extend upon.
That said, it’s been hard to decompose what it means to do research, how to do it well, and assume the traits of a good researcher. Furthermore, what does “good” even mean?
I think a good researcher is one who has their audience in mind, but I think tending to said audience can feel paralyzing. I think good research doesn’t have to stem from curiosity or “first-principles” but that being a good researcher means valuing both. I also think the best researchers have very top-down research agendas. Faculty positions will ask candidates to present research statements, which can be taken as an expression of your interests and a cover letter of past experiences but in reality read more as expressions of your research taste. Naively, I think the best of them, articulate a narrative about world order, a theory of change coupled with motivations for why that change is needed, and a set of mediums and modes for executing on that change.
Most concretely, some things I have learnt for myself are:
1) The best information is feedback. Even if you are doing theory, the practice of decomposing ideas into testable tasks is a) an exertion of skill that I’m trying to develop and b) a means of scoping out wishy-washy thoughts into ideas and then contributions.
2) Using AI is a personal choice, and it’s important you define your own criteria for what and when it’s useful (on a project-by-project basis). I am very liberal in how I use it, but a general policy is to offload tasks that are not integral to the skills I’m trying to learn. For one, I think human verification is what makes you able to qualify and claim whatever you’re putting out into the world and it’s easier to do this when you’re capable of performing the task yourself or you’ve done the task yourself and can thus replicate and explain it. AI is a collaborator and in the same way you should have a grasp of everything a human co-author contributes to your work, you should be able to do the same for an LLM.
3) I’m getting used to the fact that things are very fuzzy until a draft is being written. I’m still figuring out how to classify the projects I’ve been working on, but I think a good number have their contributions wrapped up in the presentation of data or concepts. And therefore, the methods developed at this stage are the most novel. This also means that it’s a bigger question of articulating the curiosity than “solving the problem” itself. In hindsight, this is very harmonious with my definition of what research is, but there are other kinds that inject novelty at different stages of the evolution of an exploration.
Anyways, this is just a list of writings I like. So here they are:
Making a research agenda is very good, but one thing I’m trying to get better at is formulating impact statements. Given your novel contribution, what do you expect to change? I think it’s possible to come up with empirical results and publish them without clarifying this to yourself, but it’s much more fruitful if you can clearly specify what it is that you care about.
It’s a good practice in asserting your judgments. Then, depending on what happens in the world, you can get feedback on your intuitions and overall grow as someone who is trying to build their research taste. This feedback loop is essential—it’s how you develop the ability to distinguish between work that matters and work that simply fills pages.
I guess with my affinity for writing literature reviews, I’m able to say I enjoy the empirical analyses type of work. Knowing which area of work you’re interested in producing lays some basis for the type of insight you should expect to produce. Of course, don’t let this stymie the contributions you could make.
Along these axes, there’s then a number of contributions I could make for a given work:
I do think being an RA is valuable, especially if you’re interested in the broader appeal of research. It’s also a good environment for getting a sense of the literature, especially since most of your tasks will involve reading and synthesizing it. I think it’s a decent opportunity to see what it looks like to come up with ideas and so on.
But it can definitely be a trap.
Here, I think there was an internal readjustment to orienting myself around thinking I could come up with new, important ideas. Then, it was important to test and transmit these ideas. Once you’ve developed some notion of subject-level interest, start reading the papers in a different manner: pay attention to their future work and directions that authors outline, come up with hypotheses (grapple with questions at a gears-level, that is, assert claims and intuit in what ways they may be true). Next, write these down.
Once you’re here, now it’s a great time to find mentorship. I think I’m at the point where having someone to help refine projects and plans, advise on writing papers is really nice and very valuable. Since you’ve done a lot of the heavy-lifting, the people you reach out to go from advice-givers to potential collaborators and advisors.
This is an environment where I think you can grow a lot and build fruitful mentorship. Don’t be afraid to sandbox ideas and perhaps my biggest regret is not speaking ideas into existence sooner. I think the biggest gap is going from the ideas in your mind to a workable project, and this is where a forgiving mentor who gives you direction while you explore helps a ton.
What is the value of tedium? What will change if we automate a considerable amount of mathematics and other fields? For example: what is the value of doing your own literature reviews, sketching your own proofs, and so on?
These tools can uniquely formalize a proof written on pen-and-paper, but we could head into a world where these sketches are generated by LLMs and then compiled into Lean or another formal, machine-readable representation. This makes me think of the 3-D representations with John Baez and something like could you formalize a 3-D, physical, functional, simulated representation of the world through your real-world model.
It’s more a vessel for people in other fields who already use math as a way of formalizing their domain-specific models and having an extension of that. For example, the economist wouldn’t have to solve for the integral by hand.
The tedium isn’t just busywork—it’s where intuition develops, where you build the tacit knowledge that lets you spot patterns and anomalies that automated systems might miss. The question isn’t whether we can automate these processes, but whether we should, and what we lose in translation when we do.
]]>The conversation started with me wondering what happens when underlying architectures, tools, and paradigms change — what happens to researchers and fields that depend on them? My motivation was to understand: are incumbent fields rendered useless by new approaches?
Sepp Hochreiter and Jürgen Schmidhuber came up with the LSTM (Long Short-Term Memory) in 1997. Going through Schmidhuber’s site and Google Scholar, it looks like he still publishes a very fair amount. His work now covers newer work like autoencoders, empirical research with “productified” models like GPT-4o, and architectures like Transformers.
ChatGPT responded correctly, saying that those with broad, conceptual knowledge often adapt. They understand new methods and continue to make advancements. Furthermore, it’s safe to say that LSTMs were an inspiration for their successors.
Anyway, I don’t think LSTMs were the best test-bed for my initial question. You know what really changed the game? Vectors — or rather, tensors. In some sense, we got a new vector when the tensor came around. Tensors have been around since 1898, but for machine learning, they really solidified themselves as the core representation in the early 2000s — which isn’t that long ago.
In 2009, Sutskever proposed Bayesian Clustered Tensor Factorization to model relational concepts while reducing the parameter space, and it was CNNs that really adopted tensors as the core representation for data in neural networks. Thus, future architectures took the tensor as their default representation.
That said, the uprooting of old paradigms for new ones might be especially interesting to study at the level of representation — specifically, how human-readable data is encoded. ChatGPT helped me intuit the current stack of representation, which, in today’s paradigm, can be summarized fairly clearly. (The ubiquity of Python is another interesting piece to look at.)
Even static typing isn’t really used in ML. I’ve heard of ‘RustTorch’ (or HuggingFace’s Candle) and HaskTorch projects, but I imagine the language will follow the representation — and typing alone won’t be enough to trigger a switch. Furthermore, if you’re quantizing, then you want dynamism and variance. In that case, the goal is more type tracking (for precision, etc.) rather than traditional type-checking.
That said, ChatGPT then raised an interesting case where you have an “extended type system might look like in future ML — for example, a type that tracks not just “float32” but also “quantized from float32 with ±3% error”?”
How would new representations (beyond tensors) change the stack — and what does that mean for how people program?
Today:
- Programming ML = declaring tensor operations.
- You think in terms of arrays and functions on arrays.
Future (with new representations):
- Programming ML = declaring local behaviors, graph rewrites, procedures attached to data.
- You think in terms of local programs, flows, patches, and dynamical systems — not just tensors.
Programmers might have to:
- Specify local transformations rather than just matrix ops.
- Deal with irregular structures much more naturally.
- Rely on higher-level declarative specifications (“this space should be smooth”, “this patch interacts with that one”) and let compilers figure out efficient execution.
Initially, it suggested that given the current stack, at the language-layer we would just see Python with specs for new representations.
I then asked how programming languages might change.
| Path | What Changes | Example |
|---|---|---|
| Python Evolution | Python is augmented with new types and compilers that reinterpret your code. | Like Triton and Mojo: still Python-looking, but compiles to efficient IR and machine code. |
| New DSLs (Domain-Specific Languages) | Specialized new languages designed for structure-first ML. | Like DiffTaichi for physics simulation; you describe computations differently. |
| Fully New Languages | Radical new designs for “composable dynamic systems” — replacing tensor-first thinking entirely. | No real-world examples yet at scale, but Metaprogramming Graph DSLs and functorial ML research points here. |
The language might still look like Python for a while, but the mental model you need to have changes massively.
I think this would serve as a decent basis to actually begin the research. I could imagine asking ChatGPT to conceptualize a new representation stack for the future, but given this, it would probably be too broad in scope. Your new representation stack should be grounded in some real-world motivation — for example, observing the needs of a robotics planner. Then, you would need to find a way to measure efficacy, and so on. ChatGPT could probably assist well with these steps too.
However, the last mile — intuiting how something would actually be used in the real world and polishing it into a usable product — is what I find takes the most time in any project. That said, I’m fairly sure that coming up with something like the above using only a search engine, or by going through textbooks and papers the traditional way, would have taken much longer.
Deciding that this was a path worth pursuing could be attributed to some form of “expertise.” Economists studying LLMs and research claim that this is why we haven’t yet “recursed into intellectual Narnia”: fitting concepts together remains the hard — and interesting — part. Still, it was nice to quickly get broad sweeps of existing fields without having to do tedious library work.
Like Schmidhuber, I think a major benefit of LLMs is how they can make research more enjoyable, helping eliminate a good fraction of the annoying parts. On the flip side, it seems plausible that more junior researchers like myself don’t always “pay our dues” — we do fewer reps, we get lazy. We gloss over the finer-grained patterns of the world and of our fields, ask “worse” questions, and form “worse” hypotheses. We risk operating at a level of abstraction too high to make truly meaningful theoretical progress.
That said, I’m not sure Vaswani et al. needed to know how to write a device driver from scratch to develop their work. But if they did, I’m curious what kind of tacit learning influenced their work.
]]>From conversations and assumptions, I gather they left either to make money, work on problems they felt were urgent, or because college felt unfulfilling or diminishing in return. All of these reasons resonate with me. Frankly, I left for a combination of medical reasons and, after my summer internship, not being in the right headspace to be in school. For those who ask why I didn’t apply for a transfer—this is why.
Over the past year, I’ve been fortunate to participate in opportunities that have helped me grow and feel cared for. With the spaciousness that being out of school provides, I tried to explore who I was. I forgot that being out of school isn’t a designation—people live and work in many ways as adults. I’m writing this because I think I’ve become consumed by expectations.
I value exploration, and this chapter has been about figuring out my purpose and how to orient myself to life. I believe that much of the intrinsic goodness in the world comes from our ability to make something of it. When external forces infringe on that ability, I think they’re culpable.
I’ll never get my twenties back—and I’m turning 20 next month.
Patrick Collison once wrote something I read at 17: that you shouldn’t judge your success relative to your peers. I think that still rings true. I’m a little weird relative to them, and that’s okay.
I’ve tried to go deep into different things. Most notably, I learned how to get comfortable with the terminal and the code editor. I also learned to enjoy writing—to express my thoughts—and, outside of this blog, to make those words more pointed and rooted in provoking ideas.
Domain-wise, I found a niche in policy and governance that I could speak to, albeit as a novice. I’ve grown fond of infrastructure—libraries and databases. Economics is a subject I thought I’d be good at, but I now spend little time with it. Still, I can navigate a wiki and reference the basics. Maybe I have some intuition.
Lately, I’ve felt out of breath. And when you’re catching your breath, it’s interesting how your body doesn’t do the most efficient things. Your lungs fill sharply in short intervals, your chest tightens, and your mind spirals into thinking you’ll never get enough air. You’re jailed in a loop of shallow breaths, and you forget to breathe. I feel like I’ve forgotten to breathe, but I’m trying to remember.
In responsibility, in events, in a world that feels out of control—it’s nice to feel like you have your breath and your pace. In reckoning with my values, I’ve become more confident that I care about space, choice, and the ability to act on what matters to me. I think I let that slip. This blog is a reminder to get it back.
]]>Note: I didn’t edit these questions after they were generated. Upon reading these questions, I find a lot of them pretty fluffy, but alas it took a minute max.
It was super useful and I started of by asking definition-type questions and then later asking for clarity on the assumptions that I’d made as the activity went on. I think learning how to think, write, and talk clearly about your work is very important. In fact, it’s at the root of building knowledge and disseminating it—what any researcher aims to do.
The following is the outcome of the activity: a summary of the concepts I’d learned.
We make simulations to represent complex systems. Representing systems simulationally is useful because we can interact with them in a lower-dimensional way. Simply sharing an exact-copy replica doesn’t mean that you’re sharing the right things. Categorical systems theory is a way of simulating systems with grammar. This means that if I know the grammar I can 1) rebuild the original state of the system (which is valuable to be able to do) from the lower-dim model, and 2) can understand any system that uses the grammar. Just like in natural language, you can combine thoughts into new ideas, you can do the same thing with models and thereby build new systems and worlds (which is great).
CST is a theory with two variants: one is operadic, where you model a system by capturing the whole state and can show how the state is different by looking at the graphical differences in the different captures; the second is procedural, where you model a system by capturing the transformations of the objects within it. These two variants convey different information, and each would reconstruct the original state differently.
DCST unifies the two variants into a harmonized grammar that can generalize to more simulations and helps us reconstruct original states with more confidence (as there’s a single grammar that we could add to a manual to then refer to).
]]>A few months back when I set out to really get started, a friend asked what I was reading, and I told him this book. At first glance, he thought it was pretty fluffy. I agreed at the time. Now, I would disagree — pretty strongly so. Published in 1998, his accurate predictions give him authority, and that aside, his claims resonate and the research he includes is dense but grounding. For example, he acknowledges the power of protein structure prediction but suggests that it will be solved. Of course, today DeepMind’s AlphaFold is well-known and capable. But he also awaits predictive models for environmental science. He claims that it’s harder — cells and proteins are much more deterministic than entities and organisms. And indeed, we’ve yet to crack that puzzle.
The meat of the book dives into his tranches of knowledge, including the natural sciences, the mind, genes and culture, the social sciences, ethics, and religion. He commends them for their progress since inception but ends each chapter questioning how they can be better.
He spends much of his time explaining how disciplines can improve on the basis of how strong their models are. For the most part, these models are assessed on their accuracy and their generalizability. But you’re still left asking: so what? At the end, though, you gain clarity that models are the only means in which we can reason about the world. Models are consilience.
He mentions how, with evolution, will come conservatism. The kind of conservatism that implies a narrow perspective and narrower trajectory. We’ll remedy our ailments with the new treatments we work to discover, we’ll fix our man-made problems with man-made technology. Perhaps it is that simple but he calls us to pay attention to the human ethic.
A core focus of this book is explaining how subscribing to monoliths as a scientist is stifling. A question, though, that arises while reading this book, is why? Why venture into other fields when you have your own? Well, I believe Wilson is motivated by being able to understand specific complex systems (p. 93), and doing so simply requires being able to point to different fields.
He touches on religion and ethics very beautifully and states that “Science faces in ethics and religion its most interesting and possibly humbling challenge. […] Religion will possess strength to the extent that it codifies and puts into enduring, poetic form the highest values of humanity consistent with empirical knowledge.”
In the Natural Sciences chapter (p. 53), he writes that “the theory that unites […] basic phenomena is an interlocking set of graphical representations and equations.” Here, he refers to ‘quantum electrodynamics (Q.E.D.)’ to describe the wave-particle duality (Louis de Broglie (1924)).
He also gives his takes on the nature of theory. He satisfyingly writes that complexity without reduction is art, and complexity with reduction is science (The Natural Sciences, p. 59). As I interpret this, dimensionality reduction — figuring out how to do it — is an act of rigor. Dimensionality reduction is seeking signal in the noise: it’s how we engineer understanding. And to engineer understanding is to do science.
While he does condemn computational theorists to some degree — those who make themselves victims to combinatorial explosition by simulation — Wilson writes about the value of theory (p. 56). In fact, he claims that science is nothing without theory. To theorize is to narrate. And to build a theory either from scratch or to build someone else’s is a way of reconstructing the system — to tell the story again.
“Each advance is also a prosthesis, an artificial device dependent on advanced expertise and intense continuing management.”
“A united system of knowledge is the surest means of identifying the still unexplored domains of reality. It provides a clear map of what is known, and it frames the most productive questions for future inquiry.”
“Historians of science often observe that asking the right question is more important than producing the right answer. The right answer to a trivial question is also trivial, but the right question, even when insoluble in exact form, is a guide to major discovery. And so it will ever be in the future excursions of science and imaginative flights of the arts.”
]]>“The ideal scientist thinks like a poet and works like a bookkeeper.”