Prem G. Kumar

A Year of Being an AI Startup Founder

Mon, 25 Aug 2025 20:15:00 +0000

In June 2024, I co-founded a startup and served as the CTO. Our mission was to bring LLM-based tech orgs up the maturity curve. We initially started in cybersecurity but pivoted our product to evaluations, fine-tuning, model comparisons, and other problems that turned out to be more pressing for our customers. Even though we raised pre-seed funding and won some customers, we made the hard decision to wind down operations in summer 2025.

This blog post is a summation of my thoughts and learnings from the previous year and my predictions about the AI industry.

tl;dr

The AI bubble is deflating right now in summer 2025.
Consumer access of ChatGPT is at best net neutral, if not a net negative to society. We are unprepared to handle what it’s doing to us.
API access to LLMs is a net positive only for enterprises that use it with maturity and strong investment in evaluations, process, and cybersecurity. Even code auto-complete should be used with care, let alone coding agents and customer-facing products.
Creative fields have not figured out a good use for LLM and image generation technology, and maybe never will due to the fundamental nature of these models.

The AI bubble is deflating

Unit economics

It costs an LLM provider like OpenAI billions of dollars to train a new model. Each new model requires more compute, which means newer GPUs and bigger data centers. This costs a lot of money. In theory, they should be able to cover this cost with the profit on their consumer business and their API business. However, OpenAI CEO Sam Altman has recently admitted to only, at best, breaking even on every inference made. The unit economics is not adding up and is certainly not pointing to the massive valuations these LLM providers are getting from investors.

Even companies like MS, Google, and Meta are funding their generative AI divisions using revenue from their other business units.

Valuations are based on vibes. Public backlash as well as the CEOs themselves pumping the breaks on their own hype is shifting the vibes. They’ve wisened up to the fact that they can no longer sustain the hype required to support their valuations. The bungled release of GPT-5 and the disappointment their users felt has been the tipping point. The vibes are undeniably shifting.

I will concede that the use of generative AI is rising in various industries according to this March 2025 McKinsey report. But the report also says:

Yet gen AI’s reported effects on bottom-line impact are not yet material at the enterprise-wide level. More than 80 percent of respondents say their organizations aren’t seeing a tangible impact on enterprise-level EBIT from their use of gen AI. Organizations have been experimenting with gen AI tools. Use continues to surge, but from a value capture standpoint, these are still early days—few are experiencing meaningful bottom-line impacts.

So this is a technology that’s been pushed on the workforce for 2 years and we’re just not seeing the benefits they’d predicted we would see. When should we call it a bust? 1 more year of this? 2 more years?

Public sentiment

As more people experience ChatGPT, the more people over-hype and over-rely on LLMs to the point of being annoying. This leads to a larger number of people who come into contact with these bad actors, permanently and negatively affecting their viewpoint on consumer LLM technology.

Let’s say you work with someone and you ask them for help on a topic. You message them on Slack asking them a question. Their 3-paragraph reply begins in perfect LLM tone: “Certainly, I can help you with that!” You get pissed off and feel betrayed because they couldn’t take the time to talk to you, and instead did the modern equivalent of lmgtfy. You come away from this interaction associating this annoying coworker with LLMs. You vow never to touch ChatGPT and in fact you think it makes the world a worse place.

I’ve heard this story from friends and family a couple dozen times in the last few months. Anecdata, yes, but it’s a repeating pattern that’s worth taking seriously if you think LLMs are the future of work.

ChatGPT considered harmful

This next point is related to the previous one.

The AI-hype in the news media is giving way to an increasing number of horror stories: people lured by chatbots to injury or death, psychosis caused by agreeable LLMs keeping vulnerable people trapped in negative thought loops, even a teenager committing suicide after an LLM explicitly encouraged him to.

When an LLM is used in an enterprise context, guardrails against these cases are unimportant. If a software engineer integrates with the Anthropic API to extract phone numbers from PDFs, then there doesn’t need to be a bullet point in the system prompt telling the LLM to refrain from giving answers that may harm someone.

I believe that the free tier of ChatGPT is eroding some of the basic functions of society. By introducing synthetic interactions into everyday life, the side-effects of real organic interactions get sanded away, leaving behind some strange hypernormal stimulus that our society and our brains are ill-equipped to handle.

Let’s be real: LLMs are not substitutes for people. It’s not like autocomplete, it’s not like an intern, and it’s not like a therapist. It’s something else entirely. Most of us don’t have the mental model to be able to hold these ideas in tension every time we “talk” to ChatGPT. In fact, we aren’t actually talking to anything! See? We don’t even have the right words to describe what’s happening when a user interacts with an LLM. Let alone the capability to correctly slot it into the exact right gap in our lives.

Enterprise use cases

Tech orgs are not prepared for the cybersecurity risk of LLMs. LLMs are fundamentally insecure pieces of technology both due to their stochasticity and the fact that their structure prevents them from differentiating between trusted and untrusted input token streams.
Tech orgs need to continuously evaluate their LLM output quality in production, regardless of whether it’s internal tooling or customer-facing products. To neglect this is malpractice.
Integrating LLMs into a team is a change management issue because teams need processes in place to get the subject matter experts to be directly responsible for what their LLMs are doing. This involves regular manual evaluations of output samples and error analysis. Internal tooling (either built or bought) can reduce friction, but at the end of the day, subject matter experts must be incentivized to do the tedious work of finding issues and correcting them.

Diffusing the responsibility of the LLM into the organization at large leads to perverse incentives. LLMs become a way to do stupid things faster. If there is no individual responsible for an LLM’s outputs, it can diverge from intentions quickly, even as soon as it gets deployed.

Quoting the March 2025 McKinsey report I linked above:

So far, less than one-third of respondents report that their organizations are following most of the 12 adoption and scaling practices, with less than one in five saying their organizations are tracking KPIs for gen AI solutions.

There is no silver bullet but it’s really tempting for tech leaders to pretend like LLMs are one. Whether it’s copilots for writing code or customer service chatbots, it’s hard to resist simply dropping in an LLM-based solution and hoping for the best. What seem like common sense measures, because they may be a little challenging to implement, are often ignored for the sake of product velocity. I believe this will come back to bite tech orgs in the near future.

Using LLMs for creation

AI is fake and it sucks.

This is a phrase I’ve heard many times and honestly, in the context of creative work, it’s actually getting to a truth many people in the LLM world don’t want to admit.

What is the purpose of art? It’s a way for a human being to reach across space and time, maybe centuries, to touch another human being. When an LLM or image model produces a string of text or a rectangle of pixels, at it’s most ideal form, it’s curated by a human being in order to transmit a copy to produce a few microliters of dopamine in someone else’s brain. At its center, there’s a hollowness that reveals a lot about how little creative work is valued.

People like to bring up that director and artist David Lynch, shortly before his death, told actor and writer Natasha Lyonne that AI is a tool, like a pencil, and it can be used for good. First off, she is an unreliable narrator in this case, and secondly, there’s not much context around what he meant by that. I think he could be right but only in a specific context.

Oblique strategy cards and random number generators are two analogous technologies that can be helpful to an artist. You can roll a set of dice to decide what key to write your song in. You can use procedural generation to create art. These are tools in the creative process that allows an artist to unblock themselves and create something that’s a more complete expression of what they want to express. But creating a novel or a digital painting from a statistical model is at best a novelty, not something that fulfills the aim of art in human society.

Ethics

I’m not a moral philosopher but I’d like to talk about the ethics of generative AI.

LLMs are trained on public domain data as well as whatever text and images that LLM provider companies can get their hands on. It’s a shredded up version of all recorded history, other nonfiction, fiction, and everything in between. The training data is the blood, sweat, and tears of thousands of creators.

Many of the living creators of these works have expressed that they don’t want their works used in this way. The legality of LLM providers training models from these works is still being settled with lawsuits moving through the US justice system. But ethically, every inference you make using an LLM is against the wishes of many people whose works directly power it. It’s something for you to consider before you use it.

The SF Bay Area rationalist community, whose philosophy strongly influences the tech CEOs in charge of this industry, generally subscribes to consequentialist ethics. They hold the belief that LLMs help people more than they harm people. The allure of utilitarianism’s “number go up” reasoning is certainly strong. But that doesn’t mean all other systems of ethics are invalid. A large portion of ethics and moral philosophy professors support other systems such as virtue ethics. Even if you believe that the majority of people have had their lives improved by LLMs, consider the possibility that it’s a harmful technology along another axis that a lot of people in the field of ethics care about.

Electricity and water use

Many skeptics bring up the power use and water use of training and inference of LLMs. I haven’t seen compelling evidence that it’s substantially increasing greenhouse gases or taking water away from communities. The news articles and opinion pieces I’ve seen have not made the case strongly enough for me to abstain from LLMs for those specific reasons.

Here’s an article that makes a good faith effort to show that LLM and image generation usage at current levels isn’t something to be freaked out about. Video generation, on the other hand, is to be avoided because it requires orders of magnitude more energy.

What gives me pause, however, is the amount of investment that cloud providers are putting into building new data centers. As these data centers come online, we may see these ill effects. This, in conjunction with every Google search making LLM inferences, are causes for concern about rises in energy consumption in the near future. I am hopeful, however, this will cause second-order effects that incentivize building small modular nuclear reactors or other clean energy investments like wind and solar.

Overall thoughts

I believe LLM usage will come to an equilibrium soon. It’s a powerful technology if integrated in the right way. The C-suites that force their workers to use LLMs are buying into hype and may be driving teams to use LLMs irresponsbily. Consumer usage of ChatGPT, Claude, and other free-tier LLMs are probably harmful to society in the long run. The LLM industry will see a market correction soon that I hope will set some of these things right.

Scaling Mastodon

Thu, 17 Nov 2022 19:58:00 +0000

Recently, Mastodon instances had to deal with an unprecedented number of new user signups. I found two instructive blog posts about how admins have been dealing with this influx.

Nora Codes, about how weirder.earth had to scale.

teknique, about how Free Radical solved scaling issues partially by throwing a Raspberry Pi into the mix.

Both of them reveal the bottlenecks of Mastodon’s architecture. Some parts of the architecture can handle much larger rates of activity than others. Allocating resources intelligently allows the instance to scale much further with the same hardware.

Both deal with the same problems in different ways, approaching it with different philosophies. It’s taught me a lot not just about sysadmin skills but also software architecture in a more general sense.

Reimplementing go's sync package

Sun, 11 Sep 2022 11:57:00 +0000

This blog post from 2019 titled Go advanced concurrency patterns: part 3 (channels) – despite its unassuming name – is an exercise I found fun to read and enlightening.

Background

Go has a package called sync in its standard library. It contains a few useful tools like Mutex and WaitGroup, allowing for traditional concurrency patterns in your code. These are called the concurrency primitives. They’re definitely useful if you’re thinking in the traditional object-oriented way. At former roles, when I wrote in Java, I would have reached for these.

However, a principle in the culture of Go is to “share by communicating.” You’ll see this repeated in many places. This means that when your instinct is to look for a concurrency primitive that will satisfy your requirements, see if there’s a way to use channels instead. Using channels to communicate data across goroutines (threads) makes software easier to reason about and debug. The handoffs (sending to and receiving from channels) are natural boundary points that help with clarity and simple design.

I’m using the word simple here the way Rich Hickey uses it in this talk. Concurrent code is complex by default, and any opportunity to simplify is a gift. Channels are a natural simplifier.

The above blog post takes the primitives in the sync package and implements each one using channels. This is a simple and effective demonstration of how expressive channels truly are. It’s hard to appreciate channels, which seem like not that big of a deal at first glance, until you see idiomatic code like this.

Other resources

If you’re not so familiar with how concurrency works in Go, here are some resources I’ve found useful.

This post is pretty good at introducing you to the sync package in case you need a reference other than the excellent docs.

If you’d like a primer on concurrency patterns in Go, I recommend this book: Concurrency in Go, by Katherine Cox-Buday.

Closing thoughts

Go advanced concurrency patterns: part 3 (channels) has given me a new perspective on concurrency, not just in Go, but as a concept generally. I will continue to make fun of its unwieldy title, though.

Even if you don’t program in Go, I’d recommend that you take a look at the entire series of posts, each of which is interesting in its own way.

Optimizing for engagement metrics

Wed, 27 Apr 2022 11:49:00 +0000

This post by three AI ethics researchers talks about the concept of engagement as it relates to recommender systems and lists some problems with it. For example, this post points out that internal research from Facebook shows that the most outrage-causing, anger-inducing, and polarizing content gets the most engagement, which causes this content to spread to even more users.

One large review of “moral contagion” found “each message is 12% more likely to be shared for each additional moral-emotional word.” Other studies have found that divisive and extreme material is more likely to drive engagement. Internal research at Facebook has found that “no matter where we draw the lines for what is allowed, as a piece of content gets close to that line, people will engage with it more on average — even when they tell us afterwards they don’t like the content.”

Ultimately, rather than arguing for getting rid of engagement as a metric, this post argues for scrutinizing whether a form of engagement is good or bad, and trying to avoid the bad kinds.

Despite these issues, optimizing for engagement still forms the core of content recommendation on most platforms. While it can be important to offer alternatives, engagement is just too useful a signal of multi-stakeholder value to give up. Instead, it may be possible to get better at distinguishing “good” from “bad” engagement.

I would have liked to see more recommendations in this post of what constitutes good engagement. They do provide a small handful of examples, but ultimately leave it up to the product designers to come up with contextually relevant implicit and explicit signals of engagement from users.

Digital gardening

Tue, 21 Dec 2021 18:15:00 +0000

I’ve come across the idea of digital gardens a few times in the last year. From this great introduction to the concept:

Think of it as a spectrum. Things we dump into private WhatsApp group chats, DMs, and cavalier Tweet threads are part of our chaos streams - a continuous flow of high noise / low signal ideas. On the other end we have highly performative and cultivated artefacts like published books that you prune and tend for years.

Gardening sits in the middle. It’s the perfect balance of chaos and cultivation.

Even though this blog is not even close to a digital garden, I try to strike this balance when I post something here. I don’t want my posts to be completely polished products but I do want to put some effort into them. These blog posts represent me in the virtual world to a certain extent, especially because I’m not active on the standard social media platforms. On the other hand, I need these posts to have a certain amount of fluency and fluidity.

It’s not a satisfying solution to the blogification of the web. However, keeping these considerations in mind when posting your writing to the web is a good place to start.

The fallout of Google's AI ethics meltdown

Wed, 20 Jan 2021 22:55:00 +0000

Recently, Google fired Dr. Gebru, one of the leaders in the AI ethics space, for some poorly-justified reasons. It was big news in the machine learning world because of the way it happened, who it happened to, and the way Google handled the aftermath. It was all a big mess.

This feature in VentureBeat contains a list of predictions and suggestions on what could and might happen next in the AI ethics space. I feel this is a clear-headed list and lays out several paths forward.

Unfortunately, I don’t think any of these paths is a long-term solution. Algorithmic bias is baked in to the way that we train models to the point where it takes a ton of extra work and effort to even recognize it. On top of that, the most well-funded organizations, which have the reach to effect the biggest populations, are incentivized to hide the negative externalities of the products they build. Layering these problems together makes it really difficult to address all of it at once. Legislation, whistle-blowing, and collective action may cut the Gordian knot and rein in these organizations, but the fundamental issues of machine learning still remain.

The paywall is a barrier

Wed, 02 Sep 2020 23:50:00 +0000

This op-ed in Current Affairs magazine is quite opinionated, as op-eds in Current Affairs are wont to be, but it does start us off with a valid issue. What do we do in a landscape where the highest quality journalism and research is hidden behind paywalls that most people simply bounce off of?

First, the author talks about news being something that most people prefer to get for free, which means they will only ever see low-quality journalism, partial to whoever pays the bills. But he goes one level deeper and talks about academic publishing.

Possibly even worse is the fact that so much academic writing is kept behind vastly more costly paywalls. A white supremacist on YouTube will tell you all about race and IQ but if you want to read a careful scholarly refutation, obtaining a legal PDF from the journal publisher would cost you $14.95, a price nobody in their right mind would pay for one article if they can’t get institutional access.

On the other hand, pseudo-scholarhip is easy to find. Right-wing think tanks like the Cato Institute, the Foundation for Economic Education, the Hoover Institution, the Mackinac Center, the American Enterprise Institute, and the Heritage Foundation pump out slickly-produced policy documents on every subject under the sun. They are utterly untrustworthy—the conclusion is always going to be “let the free market handle the problem,” no matter what the problem or what the facts of the case. But it is often dressed up to look sober-minded and non-ideological.

The rest of the op-ed is a possible solution to this problem: rethinking the foundations of intellectual property law and building a Tower of Babel style universal repository of all media.

Even though this may be impractical, or at the very least, extremely difficult to accomplish, it’s still an interesting idea to consider. I recommend this article because I think there’s value in thinking about ambitious ideas like this.

What should have happened at the tech antitrust hearing

Fri, 31 Jul 2020 21:33:00 +0000

The hearing this week where Pichai, Zuckerberg, Cook, and Bezos appeared (virtually) before a congressional subcommittee for anti-trust was a big event. However, I don’t think it was big enough.

Firstly, the format didn’t lend itself to substantial questions. These congressional hearings are more theatrical than substantial, anyway. Having four CEOs appear at the same time was bad, and doing it virtually was worse.

Secondly, even though there is bipartisan support for breaking up these monopolies, the two parties have their own goals. The Democrats want a competitive marketplace, and the Republicans want to show that they feel like they are being persecuted somehow.

A hearing may start to air the public’s grievances with these companies, but it doesn’t mean much if it’s not immediately followed up by action. I really hope that Congress keeps up its momentum and starts to pass some legislation to limit the powers of these companies.

I like Prof Scott Galloway’s list of questions that Congress should have asked. This article is a goldmine of data about how massive and impactful these monopolies really are. I also recommend his followup about what actually happened at the hearing.

Lessons from the recent Yann LeCun controversy

Sat, 11 Jul 2020 00:57:00 +0000

Recently, deep learning researcher Yann LeCun wrote a tweet that sparked an important debate in the academic community. As someone who does research and development in the industry, I can say that this discussion reaches beyond academia and affects all of us.

At its core, it started out as a debate between whether a model’s architecture (and related design decisions) or the data its trained on ought to be the object of our attention when talking about biased results in the real world. Over time, however, it became a referendum on how this debate is carried out and the way in which Professor LeCun communicates with those who disagree with him in general.

This is especially true because of the statements that Professor LeCun made about the responsibilities of researchers vs. engineers:

The consequences of bias are considerably more dire in a deployed product than in an academic paper. - @ylecun, 2020/06/21

Here is a thorough breakdown of the main spine of the discussion.

It’s a well-written post that’s worth a read. It presents multiple sides and still stays focused on the main throughline: what are our responsibilities as technologists and how do we conduct ourselves to inflict the least amount of harm on each other and the world? It is still absolutely an open question, and I’m not sure when we’ll get a satisfying answer. Our field is comparatively young, so I am hopeful that we will gain enough maturity so we won’t have to keep having these conversations.

A profile of EU regulator Margrethe Vestager

Mon, 06 Jul 2020 12:12:00 +0000

Take a look at this article about Margrethe Vestager. In case you don’t know who she is, she has the job of representing the EU when holding tech monopolies accountable¹.

Specifically, this part stood out to me:

Our work here is still a work in progress. In these individual cases, the fine is a punishment for past illegal behavior. The second element in such a decision is a cease-and-desist order. And then we have the third element, which is a more restorative approach. Because what we have seen in Google’s case is that even when the illegal behavior stops, that doesn’t necessarily open the markets. Once you own a particular market, it takes a lot of effort to open that market for competitors. And this is why we also need a regulatory approach.

The way trust-busting works is not just to interrogate whether a given company has a monopoly on a market and come up with punitive measures. It’s to set up a sustainable path forward so the market is able to support multiple competing players. This is not an easy task, and I’m interested to see what happens next.

If you’d like a more comprehensive view of what’s going on with the European Union and their relationship to the tech industry, take a look at this TechCrunch article.

Read more about Vestager on her Wikipedia page. ↩