Jeongkyu Shin
After Astra, What Remains for AI Companies?

Simon KimI heard that after Astra came out, you and other researchers were stunned, and started discussing it while looking through papers and prior research. That led you as far as thinking AI companies might soon go under, and I understand you held an all-hands meeting this morning. What struck you so profoundly?

Jeongkyu ShinIt didn’t feel like just another better model had come out. It made me think we might have to reconsider, from the ground up, the assumptions we’d built our business on. I kept thinking about it all weekend and couldn’t sleep properly.

As a researcher, it’s fascinating and exciting. But when you look at it as someone running a company, the question changes. If this shift is real, where does the value remain in what we’re building now? Because that’s what we have to think about.

Simon KimWhat question did you put to the team at the morning meeting?

Jeongkyu ShinIt was: how do we compete with a company capable of virtually unlimited R&D? Once models can handle much of the research and development, a company that spends more can run more experiments and get results faster. We talked about what we could bring to compete against an opponent like that.

Simon KimDo you mean that demand for AI could grow while existing AI companies actually become less profitable? We’d have to distinguish between the market’s growth and the survival of the companies in it.

Jeongkyu ShinExactly. If AI gets good enough, other people can easily build the software we’re working so hard to create now. That’s what worries me. The business could struggle not because the product is bad, but because the barrier to building that product has fallen.

Simon KimTo understand where that change begins, I think we first need to understand the industry’s structure. How would you break it down?

Jeongkyu ShinYou can broadly divide it into three layers. At the bottom is the work of optimizing computing resources and basic computation. Above that are token factories, which run models to produce tokens. At the top are agentic AI platforms that use those tokens to do work.

Simon KimDoes a competitive edge in computing infrastructure come from securing good GPUs, or from making better use of the hardware you have?

Jeongkyu ShinBoth matter. To run models, you need GPUs, memory, servers, and a network connecting the hardware. But just having the hardware isn’t enough. You need to make sure different workloads share those resources efficiently.

Simon KimAt a token factory, what operational capabilities make a difference, apart from the model’s own performance?

Jeongkyu ShinTokens are pieces of text or other data that a model processes. They don’t always correspond to a single word. One word can be split into several tokens.

The word “factory” refers to actually operating models and producing tokens. What matters here is how many requests you can handle with the same hardware, how quickly you can respond, and how much you can bring costs down.

Simon KimIf agent platforms are the layer that consumes tokens to perform real work, advances in models could increase the value of platforms while also absorbing some of their functions. Among the capabilities Lablup has built, how do you distinguish between those that compensate for model limitations and those that will remain valuable even as models improve?

Jeongkyu ShinThat’s exactly the distinction I felt we needed to revisit. Right now, a platform connects tools for finding information and reading files, turns the model’s decisions into actual actions, and then feeds the results back. Even producing a report requires a continuous process from gathering information to writing and checking it.

But if the reason we split roles across multiple models and gave them complex instructions was to compensate for the limitations of a single model, a more capable model could make that setup less necessary. On the other hand, the things you need in order to entrust it with work—permissions to access real services, execution logs, and verification of results—remain separate requirements. The problem is that the mere fact those things are necessary doesn’t guarantee that our company stands out. If other companies can implement them easily too, we have to go back and ask why customers should choose our platform.

Simon KimIf those three layers depend on one another, a change in how models compute would affect everything from infrastructure to the final product. How far across those layers does Lablup take a connected view?

Jeongkyu ShinWe work on both infrastructure and platforms, so when the computation changes at the bottom, we have to reconsider the product at the top too. Other companies aren’t confined to a single layer either. Some are building across multiple layers.

Simon KimDid your experience researching computational neuroscience make this shift feel even more significant?

Jeongkyu ShinI studied how to disentangle and understand brain circuits in which multiple functions overlap. Seeing the reverse problem—building multiple overlapping roles into a single network and having it run repeatedly—reminded me of something I used to consider difficult. Saying that the early layers extract features and the later ones organize context is also a simplification to help explain their roles. In reality, the functions of individual layers aren't so neatly separated.

Simon KimYou said you saw the potential for an internal looping architecture in Astra. How does that change the way compute is used compared with the conventional approach of making models bigger?

Jeongkyu ShinThe hypothesis we're exploring is an architecture that reuses some of its layers. Rather than continually making the model bigger, it performs more computation by passing through the same intermediate computational block multiple times.

Simon KimIf you repeat the same block instead of stacking more layers, does that also change the relationship between model depth and parameter count?

Jeongkyu ShinInside a model, the input is represented as numbers, and those numbers are then transformed through a series of steps. You can think of those computational steps as layers. A fixed-depth model passes through a set sequence of steps in a single forward pass—that is, the process of computing from input to output.

A recurrent-depth model reuses the same block. It's more like running an intermediate result through the same machine again, rather than adding a different machine for each stage of production. Having more machines and using the same machine multiple times are two different approaches.

Simon KimThen we need to consider model size separately from the amount of computation used to solve a problem. Is it possible to choose to compute for longer with the same weights instead of storing more weights?

Jeongkyu ShinIt means there's room to add computational steps without continually adding distinct weights. Weights are the numerical values a model acquires through training. But the computation doesn't become free. Running the same block again still takes time and electricity.

Simon KimI imagine the idea of repetition has been around for a while. What made it difficult?

Jeongkyu ShinIt's not that the idea itself didn't exist. The same network has to perform multiple roles over successive iterations, so training it stably is difficult. In my field, I analyzed neural circuits with overlapping functions. Here, I'm looking at the problem of getting those functions to work together within a single architecture.

Simon KimIf the same circuit has to play a different role on each iteration, it must also be difficult to meet the training objectives. Is the key to make sure those iterations lead to useful computation?

Jeongkyu ShinThat's part of it. There's no point if the information you need disappears or the results get worse with each iteration. You have to train it so that the computation continues to be useful. When we say training converges, we mean that training progresses toward stable errors and behavior.

Simon KimSo the surprise isn't so much the idea of a loop as the possibility that they've made it work at a high level of performance.

Jeongkyu ShinYes. If that assumption is right, we need to revisit the very reasons we thought it wouldn't work. But we're still curious about exactly how they trained it, too.

Simon KimExisting reasoning models also spend a long time thinking. How is this different from chain of thought, or CoT?

Jeongkyu ShinWe're looking at the difference between expressing intermediate reasoning as tokens and using those tokens in the next computation, versus performing repeated computation internally. If more computation happens internally, the model may need to generate fewer tokens externally.

Simon KimSo the important distinction isn't whether the reasoning appears on screen, but whether the intermediate steps are expressed as tokens. What changes when computation happens in latent space?

Jeongkyu ShinExactly. Turning intermediate reasoning into tokens and showing those tokens to the user are separate things. There can be reasoning tokens that never appear on screen.

Latent-space reasoning is an approach that continues computation using internal numerical representations that haven't yet been turned into words or sentences. The distinction is between writing out a solution in sentences and doing more computation on representations before they're put into sentences. That doesn't mean it's exactly the same phenomenon as thought inside a human mind, though.

Simon KimAs internal iteration becomes more capable, how should its role be divided with the external agent loop? Do the two architectures replace each other, or can they work together?

Jeongkyu ShinThey can work together. An agent loop repeatedly calls the model, runs tools, and checks results from the outside. Internal iteration reuses a particular computational block within a single model execution. The location and role of the repetition are different, so having one doesn't necessarily eliminate the other.

Simon KimThen a competitive edge would come not just from the ability to iterate, but also from judging when to stop. Is there any guarantee that computing for longer leads to a better answer?

Jeongkyu ShinThere's no such guarantee. Additional computation helps only within a certain range, and too much repetition can actually hurt performance. Deciding when to stop also remains an open problem. Having an architecture that can iterate and being able to recognize the right answer and stop on its own are different capabilities.

Simon KimIf the iteration depth isn't fixed, that could change not only how we improve performance, but also the interfaces that let us intervene while the model is thinking.

Jeongkyu ShinI've imagined that, too. If we could ask for a progress update or add constraints while computation is underway, we might not need to call the model repeatedly from the outside the way we do now. Of course, having a loop doesn't automatically give you that kind of interface. I still don't know whether Astra can actually do this, or whether there's no limit on the number of iterations.

Simon KimIt must also matter how long the leading company remains the only one that knows how to implement it.

Jeongkyu ShinIt reminds me of how DeepSeek's public research followed the release of o1. If a team like DeepSeek or Kimi publishes a similar approach, the impact could spread further. I think we might see something in about half a year. It's not that I know of any set timeline. I suspect large-scale compute and high-quality data may have made the training possible, but that's just my guess, too. We won't know how they actually trained it until they disclose it.

Simon KimWhy does more internal computation and fewer tokens matter for businesses?

Jeongkyu ShinRight now, a lot of businesses revolve around how many tokens they process. But if producing answers of similar lengths can actually require vastly different amounts of computation, we need to rethink the relationship between token volume and cost.

Simon KimSo a user might receive a short answer even though the server spent a long time computing it. How should we handle the gap between pricing based on output length and the actual underlying cost?

Jeongkyu ShinRight. The answer can be short even if the computation takes a long time. It becomes harder to infer the amount of internal work from the length of the generated text alone. Providers could measure and charge separately for reasoning tokens or computation time. So I don't think token-based pricing will necessarily disappear.

Simon KimWill it also become harder to predict how long customers will have to wait?

Jeongkyu ShinThat's another important issue. If the number of internal iterations varies from request to request, the time it takes to produce an answer can vary, too. We need to rethink how existing serving systems schedule those workloads.

Simon KimIf the amount of internal computation varies by request, increasing GPU utilization could conflict with reducing customer wait times. What needs to be adjusted on the serving side?

Jeongkyu ShinFirst, you have to coordinate which machine each request goes to and which requests to batch together. You also need to allocate memory and handle failed requests. All the work involved in running a trained model so it can respond to real users is called serving.

It's frustrating for users if short requests keep waiting behind one request that takes a long time to process. When the amount of internal computation varies, managing those queues becomes important too. You have to look separately at the time to the first response, the generation speed after that, and the time it takes to complete the entire task.

Simon KimSo far, we've optimized prefill and decoding separately. Could looking at the nature of the work in each stage help us understand the assumptions that existing infrastructure was designed around?

Jeongkyu ShinYes. Let's follow a single request from the moment it arrives until the answer is complete. We can use a typical autoregressive transformer with a KV cache as our reference. I don't know Astra's undisclosed architecture either, but I can explain how existing models generate text.

Suppose a user asks, “Read this report and summarize the key risks.” The service assembles an input that includes not just the question, but also the system instructions, any necessary conversation history, and the report. It converts that into token IDs and assigns the work to a server. If the server is busy, there will be a wait as well. We don't call this entire preparation process prefill.

Prefill comes next. As the model processes the input tokens, it computes internal representations and stores keys and values in the KV cache for use during subsequent generation. The cache stores intermediate results so the model doesn't have to repeat all the preceding computation from scratch every time. Since the input is already available, computation at many positions can be handled together. But there are still constraints that prevent each position from seeing future tokens, and computations across layers still have to follow a particular order.

The output from the final position in prefill is used to calculate the probabilities for the first new token, and one is selected according to the generation rules. If prefill is the stage where the model reads the report, this is where the first piece of the answer is determined. It's not as though the model has already composed a complete answer in its head and is simply releasing it a little at a time.

Simon KimSo decoding isn't the process of revealing a predetermined answer on the screen. It's recalculating each time what to write next, following the answer generated so far. Does it also recompute the entire report each time?

Jeongkyu ShinIt reuses the keys and values from the input it has already processed. The token just selected is fed into the model, which uses the previously stored information to calculate the probabilities for the next token. The new token's key and value are added to the cache too. The token selected this way then becomes the input for the next computation.

That repeated process is decoding. In standard generation, the preceding token has to be determined before the next one can be computed, so there is a sequential dependency within a single answer. But that doesn't mean the entire GPU processes only one token at a time. Requests from multiple users can be batched and run together, and there are optimizations such as speculative decoding, which proposes multiple candidate tokens and then verifies them.

Generation ends when an end token appears or a condition is met, such as a configured length limit or the user stopping the response. The service converts the generated tokens into text and delivers it, then frees up any per-request memory that is no longer needed. With streaming, it starts sending portions of the content while generation is still underway; otherwise, it collects the results and sends them together. The chunks of text you see on the screen may not correspond exactly to the tokens the model processes internally.

Simon KimSo when a user feels that “the response takes too long to start” versus “the response comes out slowly,” those are different problems. What timing metrics should we look at for each?

Jeongkyu ShinThe time it takes to produce the first token is called TTFT, or Time to First Token. Depending on where measurement starts and ends, it can include request transmission and queuing, input preparation, prefill, and selection of the first token. So you shouldn't immediately conclude that prefill is slow just because TTFT is long. For models that generate reasoning tokens first, there can also be a gap between the first internal token and the first answer the user sees.

Once generation starts, we look at the interval between tokens, or ITL, and the number of tokens generated per second. ITL stands for Inter-Token Latency. For a request that supplies a long report and asks for a short answer, input processing may account for a large share of the work. If a short question prompts a long piece of writing, the cumulative decoding time can be substantial. The total time the user waits includes both stages, along with queuing, transmission, and so on.

Simon KimThen when you ask follow-up questions about the same long document, does prefill have to start from scratch every time? I'm also curious about where caching changes the cost structure.

Jeongkyu ShinIf the server is implemented to reuse the KV cache for an identical input prefix, it can reduce computation for the portion it has already processed. This is called prefix caching. Newly added input still has to be processed, and conditions have to be met: the cache must still be available, and the model and input must match. Just because earlier content is visible in the conversation doesn't necessarily mean the server still holds that cache.

Keeping a cache around longer reduces recomputation, but it occupies memory. You have to weigh the benefits of reuse against the space needed to handle more requests simultaneously. Ultimately, you have to manage the whole process together—from receiving requests through prefill, decoding, cache management, and completion—to understand the actual costs and response speed.

Simon KimWhy do input processing and token generation have different bottlenecks even when they run on the same GPU? Should we think of it in terms of differences in the amount of computation and data movement?

Jeongkyu ShinA bottleneck is the part that most constrains the overall speed. To use a kitchen analogy, you might have plenty of ingredients, but cooking takes a long time. Or the cook might be fast, but the ingredients arrive late.

Prefill processes a lot of input together, so compute capacity often becomes important. Decoding generates tokens sequentially, so the speed at which the necessary data can be fetched from memory often becomes important. We call the former compute-bound and the latter a memory-bandwidth bottleneck. But it depends on factors such as batch size and model design, so it isn't an absolute distinction.

Simon KimSo you're saying that running loops internally could also shift where the bottleneck is.

Jeongkyu ShinYes, because it changes which computations are repeated and which data is read again. We have to start by revisiting what the existing optimizations were designed to speed up.

Simon KimTo understand the idea that reliance on HBM could change, I think we need to distinguish memory capacity from bandwidth. Which one limits performance in conventional inference?

Jeongkyu ShinHBM stands for high-bandwidth memory. It's used to supply large amounts of data quickly to a compute device. Here, we need to distinguish capacity from bandwidth. Capacity is how much it can hold; bandwidth is how much it can move in a given amount of time.

Even if a warehouse is large, ingredients come out slowly if the doorway is narrow. No matter how fast the GPU is, it has to wait if data arrives too slowly. That's why memory speed can have a major impact on overall performance.

Simon KimIf the same weights are used repeatedly, does that also reduce the amount of data read from memory? What conditions have to be met for reuse to provide a real benefit?

Jeongkyu ShinThere may be opportunities to reuse weights, but you can't draw that conclusion just from the fact that there's repetition. It depends on which memory the data resides in, how much of it has to be read again, and how much intermediate state is stored. Storing fewer weights and moving less data are also two different things.

Simon KimThe KV cache is central to reducing recomputation in conventional inference. Why does a looping architecture change what needs to be stored and reused?

Jeongkyu ShinConventionally, you store the keys and values generated for previous tokens during attention computation and reuse them when generating the next token. In a looping architecture, you pass through a block multiple times, so you also have to think about which iteration's information to reuse.

Simon KimTo understand that difference, we should look at how the lookup criteria interact with the stored information in attention. Of the query, key, and value, which ones are reused?

Jeongkyu ShinThe keys and values are what you store and reuse. Attention calculates which pieces of previous information to draw on at the current position, and how much weight to give each one. The query represents the criteria for the information you're looking for now, the key is the representation compared against those criteria, and the value is the content retrieved and combined based on that comparison. These are all numerical representations inside the model.

You could compare it to preparing an index and reference information in advance so you can find the parts you need in a book. But it doesn't literally store sentence summaries or bookmarks.

Simon KimThen a cache that reduces model computation and long-term memory that preserves work context need to be designed separately. Progress in one doesn't replace the other, right?

Jeongkyu ShinRight. A KV cache is meant to reduce model computation, while an agent's long-term memory is a mechanism for reusing work records or user information later. Both are also different from a response cache, which stores a completed answer and serves it again as is.

Simon KimWhat trade-offs are involved in storing the state of each iteration separately versus sharing it?

Jeongkyu ShinThe memory and computation required can vary depending on whether you keep each iteration's state separately or share some of it. What to retain and reuse becomes a design question.

Simon KimDo you think these changes could pose a greater challenge for AI chips other than GPUs?

Jeongkyu ShinMy thinking is that a chip heavily optimized for an existing computational flow could be at a disadvantage if that flow changes. GPUs have programming flexibility, so there's more room to adapt. Designs that gained efficiency by tailoring themselves to a particular approach may need to be revisited.

Simon KimCan we judge the advantages and disadvantages simply by classifying chips as GPUs or NPUs? It seems we should first acknowledge that each product has a different design.

Jeongkyu ShinA GPU is a device that performs many computations in parallel, and it's now widely used for AI computation as well. NPU is a broad term for accelerators specialized for neural network computation. TPU is a family of accelerators made by Google. You shouldn't assume that all chips have the same architecture or capabilities just because they're called NPUs.

Simon KimBeing able to run a new architecture and being able to run it cost-effectively are different things, then. In practice, does adaptability come down to hardware or software?

Jeongkyu ShinBeing able to run it and being able to run it efficiently are different things. You have to look at the compiler that translates model operations into machine instructions, the runtime that manages execution, and the memory and control architectures together.

For example, a systolic array is a structure in which small processing units perform matrix computations while passing data to their neighbors. It's like a regular, orderly production line. But just because the production line has a regular internal structure doesn't mean the factory as a whole can't repeat a process.

Simon KimSo a design optimized for today's models could turn into a cost of adapting to change when the next model arrives. Does that mean we need to consider not only a chip's performance, but also how cheaply it can support new operations?

Jeongkyu ShinYes. My concern is that the scale of that change could be substantial.

Simon KimWhich companies specifically did you see as exposed to these changes?

Jeongkyu ShinRebellions and FuriosaAI in Korea came to mind, along with Cerebras, SambaNova, and Google's TPUs overseas. I'm very concerned, but their designs are all different, so I can't say that none of them can execute loops. And I don't know every product's control flow, compiler, or memory architecture. What worries me is whether the efficiency gained by tailoring a design to a specific computational pattern will hold up under a new architecture.

Simon KimYou also connected the discussion around NVIDIA's CPX to this change, right?

Jeongkyu ShinYes. Looking at which stage of inference a chip is designed for, and how its memory and inter-device connections are configured, can give you a sense of what kinds of computation will matter in the future. As I followed discussions about the product direction, I wondered whether they had anticipated changes on the model side.

Simon KimWe should distinguish between data movement within a GPU and communication between GPUs, then. What bottlenecks are addressed by changing the memory versus changing interconnects like NVLink?

Jeongkyu ShinYes. You have to distinguish the speed at which a GPU retrieves data from its own memory from the speed at which multiple GPUs exchange data with one another. NVLink and NVSwitch are technologies that help GPUs exchange data. Those connections become important when an entire model won't fit on one GPU, or when multiple GPUs work together.

Simon KimWas there something you'd heard beyond the publicly announced specifications that brought CPX back to mind?

Jeongkyu ShinI heard that a plan that had been halted was being revived in a form that uses HBM. That made me wonder whether NVIDIA had known in advance about this shift in how computation would work. But I don't know whether they actually changed the plan or knew about it beforehand. The initially announced Rubin CPX specifies 128GB of GDDR7. The HBM story is something I heard secondhand, and the connection to changes in model architecture is my own speculation.

Simon KimLablup is already bringing multiple agents together for development and automation. Do you think that setup will be affected too?

Jeongkyu ShinYes. We've built a setup that distributes work across multiple models and ties them together for automation. But if a single, much stronger model handles complex tasks better, we have to reconsider how much value remains in the division of labor we worked so hard to build.

Simon KimSplitting work across multiple agents gives you the benefits of parallel processing, but it also creates coordination costs. In your current division of labor, what do you split up, and how do you reconcile the results?

Jeongkyu ShinFor example, you might have one agent write code, another test it, and a third review the results. You can also split tasks up and process them simultaneously. But passing results between agents and coordinating them introduces costs and errors too.

Simon KimEven as a single model becomes more capable, parallel execution and separating permissions may still be necessary. Which kinds of division of labor will remain, and which will diminish?

Jeongkyu ShinDividing work for parallel processing, separation of permissions, or independent verification could still be useful. On the other hand, tasks that were split up in complicated ways to compensate for a model's limited capabilities could be brought back together. It's not that we no longer need multiple agents. It's about revisiting why we divided the work in the first place.

Simon KimSo we need to distinguish between structures built to compensate for a model’s limitations and structures that are essential to the actual work. Do you mean that as models improve, we should scale back the former and reassess whether the latter is what gives the product its value?

Jeongkyu ShinYes. That’s why we need to rethink not just the technology, but what differentiates the product.

Simon KimThat must have a big impact on the combination of models you’ve been running, too.

Jeongkyu ShinInternally, we’ve been running a farm that distributes work across multiple instances of models from the Kimi and Qwen families. We organized that into a workflow called Squad. We turned collaboration between multiple models into a product, but if a single model can outperform that setup, the cost of coordinating them could start to look wasteful. We need to reassess what differentiates our product, too. We also see the focus of optimization potentially shifting toward faster iteration or accelerating attention.

Simon KimIt’s good news if you look only at your own team’s productivity, but the story changes if competitors can build just as fast. Do you think software companies will retain the benefits of those cost savings, or will they pass them on to customers through lower prices?

Jeongkyu ShinIt’s great as long as we’re the only ones using it well. But if other companies can build similar things with the same tools, improving productivity and becoming more competitive as a business are two different things. The optimization we do now is also a cycle of forming hypotheses, experimenting, and measuring. If that process gets dramatically faster, the resources we need could change fundamentally.

Simon KimSo a company might get better at building things while losing its pricing power. In that case, shouldn’t the first question be why customers would still pay, rather than how much you’ve cut development costs?

Jeongkyu ShinExactly. Lower technical barriers to entry make it easier for new competitors to enter. But that doesn’t mean security, understanding the customer’s work, day-to-day operations, and incident response all go away. We need to identify which parts customers will keep paying for.

Simon KimUltimately, if AI can design and implement a platform itself, we’re left asking: who is the customer of a platform company?

Jeongkyu ShinThat’s what I spent the whole weekend thinking about. I’ve long believed this change would come someday, but once it actually feels close, it becomes a management issue.

Simon KimThe optimization Lablup has been doing ultimately comes down to forming hypotheses and running experiments, too. If the cost of that drops sharply, will the value of your accumulated know-how change as well?

Jeongkyu ShinWhat matters is how close you can get to the hardware’s theoretical performance limit. The Roofline model lets you examine whether compute performance or memory bandwidth is setting that limit. It’s a way to find bottlenecks by looking at how much data you need to move relative to the amount of computation. We’ve been approaching those limits by distributing experiments and comparing the results.

But if we can do the same work with far fewer resources, we have to reassess our current development investments, too. It’s enough to make me wonder whether, two months from now, we might need only one-tenth of the resources. That isn’t a figure based on actual measurements. The concern is that if costs fall that quickly, things could change before we even recoup the money we’re investing now.

Simon KimI’ve also thought that AI infrastructure could become like electricity. Right now, enormous amounts of money are going into the infrastructure race to serve businesses and developers. But I felt that as that race matured, what ordinary people actually use it for would become more important.

Technology is advancing so much, yet booking a single restaurant in Asia still isn’t a seamless experience. So I thought our role should be less about building frontier models ourselves and more about bridging those gaps and making the technology work well in everyday life.

Jeongkyu ShinYou’re saying there’s a gap between a model’s intelligence and the experience of connecting it to real services.

Simon KimYes. Giving a restaurant recommendation and completing a reservation are different things. You need to follow through on checking availability, getting the user’s consent, confirming the booking, and handling changes and cancellations. Without permission and a way to access the actual service, even a smarter model will hit a wall.

An API is a way for a program to use a service’s features, but you also need to connect those pathways and handle failures. For consumers, what matters is whether their task actually got done, not the model’s score.

Jeongkyu ShinIt must also be important to deliver that within experiences people already know.

Simon KimPeople don’t use KakaoTalk or LINE because they compare features every time and pick the best one. They use them because their relationships are already there and they’re in the habit of using them. Those everyday experiences and connections are what we’re looking at.

Jeongkyu ShinI used to compare AI to electricity, too. But now I’m starting to think cars might be a better analogy. They all provide transportation, but the price and experience vary greatly from car to car.

Simon KimSo even as the barrier to accessing AI falls, there could still be a gap in people’s ability to buy enough compute and fast responses. Are you saying widespread availability and equal access are different things?

Jeongkyu ShinYes. Gaps could remain depending on how much compute you can use and how quickly you can get results. If you spend more, you can do more research and development, and you can pay for faster responses, too.

Simon KimEven with the same model, one side might run a small number of experiments sequentially while the other runs many experiments at once.

Jeongkyu ShinExactly. Being able to use a model and being able to use it at sufficient scale are different things. That’s why we end up asking, ‘How do you compete with a company doing unlimited R&D?’

Simon KimMoney can buy more experiments, but deciding which experiments to run and whether the results are valid still requires separate judgment. To compete, should smaller companies look to differentiate themselves through that judgment and their own unique data?

Jeongkyu ShinYou need to choose the right problems, have data, and have criteria for evaluating the results. The ability to apply them in the real world matters, too. ‘Unlimited’ doesn’t mean those constraints disappear. It means the scale of experimentation a competitor can bring to bear could become overwhelmingly large.

Simon KimIf AI becomes good at semiconductor design, too, production will be the next challenge. Even with a design in hand, you need a factory to make it.

Jeongkyu ShinThat’s right. Software changes quickly, but concrete is slow. Building factories and data centers and connecting them to power doesn’t change at the same pace. That difference in speed keeps holding things back.

Simon KimDo you mean that even if the cost of using intelligence falls, actual prices could remain high if equipment and power are in short supply?

Jeongkyu ShinExactly. You can send a good design file in an instant, but you can’t send a factory. Manufacturing, validation, equipment installation, and power supply each take their own time. Just because the digital world speeds up doesn’t mean physical supply immediately catches up.

Jeongkyu ShinI’m also reminded of the experiment involving a fruit fly’s brain connectivity. Seeing examples like that makes me rethink the pace of change.

Simon KimThat’s the case where they ran a simulation based on a map of a fruit fly’s neural connections and had it move in a virtual environment. What impressed me was that, with AI’s help, people are even attempting to turn complex biological structures into executable programs.

Jeongkyu ShinHere, a brain wiring map, or connectome, is data showing which neurons are connected to which other neurons. The field that studies those connections is called connectomics. If you turn that network into a computational model, feed it sensory input from a virtual body, and translate neural activity into movement, you can observe how it responds.

But that doesn't mean AI designed a fruit fly's brain wiring map from scratch. Using existing connectivity data obtained through experiments is also different from reproducing all the functions of a living brain. Having a road map doesn't mean you know how traffic flows at different times of day. In the same way, you also need models of the conditions under which neurons respond and how they transmit signals.

Simon KimThen we should distinguish between the research achievement of mapping the network and the achievement of building a simulation that makes it work. Still, it's a big deal that the pace of drawing on data from different fields, implementing it, and experimenting with it is changing. In semiconductors, too, it could speed up not just the process of proposing designs, but the cycle of simulating and revising them.

Jeongkyu ShinSeeing those possibilities is what led me to think about semiconductor design. Of course, a single experiment with a fruit fly's neural network doesn't prove that chip design automation is a solved problem. With semiconductors, you have to satisfy power, performance, and area constraints together, and go through design verification. Still, I felt we needed to ask again just how much faster the process of forming hypotheses, testing them computationally, and revising them could become.

Simon KimIf AI gets to the point of designing and optimizing semiconductor architectures more efficiently than experts, wouldn't that take 2–3 years at most?

Jeongkyu ShinI think your timeline is too long. I think it'll be done in 6 months.

Simon Kim6 months? Then the bigger issue won't be design, but when the factories will be ready to manufacture chips based on those designs.

Jeongkyu ShinRight. The timeline I just gave is my forecast for design automation. It doesn't mean the entire process of manufacturing and supplying chips will be completed within that time. Even if design gets faster, you can't build factories, data centers, and power grids at the same speed. You get a gap between the speed of software and the speed of concrete.

That's also why I think of examples of building data centers quickly with temporary structures, or supplementing the power supply with on-site generation and batteries. I've heard stories about Meta and Manus, and I've also heard that xAI's Colossus changed its batteries and power-generation equipment. I don't know exactly why they did that in each case, or on what scale. But what's clear is that you need not just enough power, but also the ability to withstand sudden fluctuations in load. Even if we get faster at creating intelligence, expanding the actual supply still takes physical time.

Simon KimHave there been other moments when you had to make major decisions as a company? Have you pivoted several times?

Jeongkyu ShinWe've always been building an AI platform. Rather than completely changing the business, there were moments when we decided where to focus within it. It started as Sorna and later became Backend.AI.

There was a time when we were trying various things, including a consumer AI cloud and a coding platform. But we didn't have a large team, so we decided to focus on scaling infrastructure.

Simon KimBy focusing on infrastructure scalability, you were choosing a problem that couldn't be solved just by adding more hardware. What operational capabilities become important as workloads grow?

Jeongkyu ShinWhat's important is keeping things running without bottlenecks while multiple workloads share resources. As workloads grow, you need to scale up resources too, but adding more hardware doesn't automatically give you efficiency.

Simon KimWhat was the next big change?

Jeongkyu ShinThe release of Stable Diffusion was another major shift. Before that, our customers were mainly training models. Then the inference market—actually running models to produce results—became important. We were seeing the center of gravity shift from building models to using them.

Simon KimWhen your focus shifts from training customers to inference customers, the operational requirements must change even if you're working with the same models. What changes the most?

Jeongkyu ShinTraining is the process of adjusting a model's weights using data, while inference is using that trained model for real questions or tasks. Supporting a model research team and reliably handling requests from large numbers of users involve different customer needs and different ways of operating.

Simon KimI understand you also received a proposal to become a neocloud provider.

Jeongkyu ShinSomeone offered to provide the capital for us to move in that direction, but we ultimately declined. Revenue could grow, but from my perspective, it felt like we'd be moving closer to an asset-leasing business, and I had doubts about profitability too.

Simon KimIf you became a neocloud provider, profitability would depend not only on how competitive your software was, but also on securing hardware and keeping utilization high. How did you assess the capital requirements and operational risks?

Jeongkyu ShinA neocloud is a cloud business specializing in providing AI compute resources. Building software that helps customers use their hardware efficiently is different from acquiring and supplying the hardware ourselves, both in the capital required and the risks we'd have to bear. What matters then is how consistently you can keep that expensive equipment running once you've bought it.

Simon KimHow big is the team sharing the responsibility for these decisions?

Jeongkyu ShinWe're a team of 49. That includes two people working overseas, in business and front-end roles. With limited staff, we have to decide where to focus. When technology changes, the organization immediately has to grapple with that question too. I was sharing that sense of urgency with the team just this morning.

Around 2017–2019, we saw companies around us drop AI and move into blockchain. We stayed the course as an AI platform, but scaled back our various experiments, including the consumer cloud and coding platform, and focused on scaling. That was also when we changed the name from Sorna to Backend.AI.

In 2022, when it became possible to run Stable Diffusion on a personal GPU to generate images, I really felt the shift from a market for training models to a market for using them. I got deeply into image generation myself, and that interest extended into video work. Last year, I spent about two months considering the neocloud proposal before turning it down. Regardless of the potential for a big increase in revenue, I was concerned about how much profit would remain in a business centered on leasing assets. Now I'm asking a more fundamental question than I was then—about the software business itself.

Simon KimCould you also show us the agent platform you're building?

Jeongkyu ShinIndividual agents can work with files or the screen on a desktop, and you can group multiple agents into a team and assign them work. You set permissions, configure what should run automatically and when, and organize workflows into squads.

Simon KimAre squads less about adding more agents and more about designing the sequence of tasks and the handoffs between them? How do you put an actual workflow together?

Jeongkyu ShinYes. You're linking the work of multiple agents into a pipeline. For example, if you connect research, drafting, and review, you define what each stage receives and what it passes on to the next stage.

Simon KimCan you also import things you use in Claude Code?

Jeongkyu ShinI can show you features for importing skills or downloading community-created resources. There are also OpenClaw and Hermes integrations. We've integrated some things separately as well, beyond what's visible in the version I'm demonstrating now.

Simon KimIf you can import skills, you could reuse another team’s way of working without retraining the model. Is the key that you’re sharing procedures rather than knowledge?

Jeongkyu ShinSkills are a way of packaging procedures, reference materials, and so on for a particular task so they can be reused. They let an agent draw on instructions for how to do the work, just as a person would consult a work manual.

Simon KimBut I feel like I’m seeing similar features in other tools, too.

Jeongkyu ShinThey really are becoming more alike. People have similar needs. To delegate work, you need instructions and reference materials, and you need to track who did what and when. We started with the model runtime, but we’re heading toward a similar place as tools that started elsewhere.

Simon KimThe longer you delegate work, the more the ability to retrieve and reuse records and supporting evidence matters, rather than the quality of any single answer. How do you connect memory with source materials?

Jeongkyu ShinRight. If the user has to explain the previous work every time, that defeats some of the purpose of delegating it. You need to keep a history of the work and be able to retrieve the information you need. You can also keep the source materials used in research and make them available for people to read as a wiki, or for agents to find and use through tools.

Simon KimSo the stored materials can’t just sit in a wiki for people to read; agents need to be able to use them to take action. What role does tool calling play in that connection?

Jeongkyu ShinWhen the model requests that a file be read or a search be run, the platform executes that function and returns the result. You need that connection to move from simply generating answers to actually getting work done.

Simon KimIf customers’ needs are the same and the intelligence being built is similar, features will converge quickly. How, then, do you distinguish between investing in rebuilding the same features and investing in an experience your own customers truly need?

Jeongkyu ShinThat did make us wonder whether we really needed to build this ourselves. But we believe we also need the platform on top to connect everything from the underlying infrastructure to the end-user experience, so we’re continuing to build it.

Simon KimIf what customers are buying is completed work rather than performance at any individual layer, is your conclusion that you need to connect everything from the underlying infrastructure to the user experience?

Jeongkyu ShinYes. Because what customers need isn’t just optimization at a particular stage, but an experience that ultimately gets the work done. You start to see resource management, model serving, and token production underneath, and the agent platform on top, as connected.

Simon KimIf an infrastructure company builds a complete product for getting work done, it ends up competing with its existing customers. Conversely, if an app company moves down into the lower layers, couldn’t today’s supply chain turn into a relationship where everyone is taking customers from one another?

Jeongkyu ShinThat’s right. When infrastructure companies build complete work products and app companies directly optimize model operations and costs, their boundaries can overlap. Companies that used to occupy different layers end up competing over what they can offer the same users.

Simon KimIf anyone can build similar features with the same model, why would customers choose us? If technology is quickly replicated, where do you think a company’s remaining differentiation comes from?

Jeongkyu ShinRight now, I’m thinking that perhaps brand is what remains. Even when the underlying technology in cars is similar, people choose different brands and expect different experiences. So even if you use similar intelligence to build similar features, people’s final choices may not be exactly the same.

Simon KimWhen you say brand is what remains, is that less about getting your name out there and more about giving users confidence in how much work they can entrust to you? As features become more alike, accountability when things fail and a consistent experience could become the deciding factors.

Jeongkyu ShinI think it ties into those issues. What level of quality users can expect, how much work they can entrust to you, and who takes responsibility when something goes wrong will all matter. Promises alone won’t give you a competitive edge. People will need to build up actual experience using it.

Simon KimOn the consumer side, I think that point of contact is messaging. People keep using KakaoTalk or LINE because their relationships and habits are there. I see an opportunity to rebuild that experience around agents.

Jeongkyu ShinThe existing messaging companies are probably thinking along those lines, too. But they already have established brands and user expectations, so changing won’t be easy.

Simon KimI think adding AI to an existing interface is different from building an experience around delegating work to AI from the outset. A new service can set different expectations right from the start.

Jeongkyu ShinSo familiarity can be both an asset that brings users in and a constraint on change. It can be a lot to ask of someone who came to exchange messages to suddenly take on a different role.

Simon KimRight. On the other hand, even if a new service is good, getting your friends and colleagues to move over is hard. So I’m not saying existing services are bound to fail or new ones will automatically succeed. I think it comes down to what new expectations you create and whether you actually meet them.

Simon KimIs it user expectations that make it hard for existing messaging services to act, even when they recognize the changes underway?

Jeongkyu ShinI heard from someone at Kakao that they’re preparing an agent-centered overhaul, aiming for the first half of next year. I don’t know whether that’s an official timeline. Even changing the home screen meets resistance. I also heard that ChatGPT in KakaoTalk is getting less use than expected, though I don’t know the exact usage figures myself. Looking at Kanana, too, I felt that changing the experience users expect wouldn’t be easy.

Simon KimAdding a feature is different from getting people to learn new behaviors while preserving their existing relationships. That’s why I’m focusing on agentic messaging. What existing services already have could actually constrain their ability to change.

Jeongkyu ShinThink about Facebook and Instagram. You can’t merge their user experiences and brands into one just because they belong to the same company. People expect different things from each. With cars, too, even if they use a shared foundation from companies like Huawei or Foxconn, the brands and experiences people choose can differ. Let’s say an Apple Car comes out. Is someone who buys it buying a car, or are they buying Apple?

Simon KimListening to you today has clarified something I’d been vaguely sensing since the weekend. It doesn’t stop at models getting better; it makes you reconsider the companies built on top of them and the competitive landscape itself.

Jeongkyu ShinAs a researcher, I find it fascinating and amazing. But there’s also the part of me that runs a business. That side is in emergency mode. I’m rethinking whether the assumptions we’ve been working with still hold.

Simon KimSo ultimately, you’re not worried that AI will no longer be needed. You’re worried about which parts of what you’ve been providing will no longer be scarce as AI improves.

Jeongkyu ShinYes, that’s what worries me. Even if implementation gets easier, the challenges of actual operations, customer experience, and access to resources will remain. And you have to decide which of those things to keep building yourself and which to leave to other technologies.

Simon KimI’m also curious why you felt this shock goes beyond a question of strategy for a single company.

Jeongkyu ShinIt’s like the moment when the climate crisis stops feeling like a warning about something that will happen someday and starts feeling like a reality you already have to respond to. Something you’d only known as a possibility becomes a problem that demands a decision right now. That’s how I feel at the moment.

It also reminds me of the days when we imagined what would change once every home was connected to the internet. We talked about how reading Bill Gates’s book led me to choose computer science. Now I find myself imagining what will change when intelligence becomes widely available. That’s also made me more interested in writing science fiction and other fiction. It’s about taking the changes we can see now and imagining the world that comes next. Of course, not everything will turn out the way we imagine.

Simon KimSo you’re saying you feel both excitement as a researcher and a sense of crisis as a CEO.

Jeongkyu ShinFinding something fascinating and fun is different from being responsible for a company. We’re in emergency mode as we think this through. And when I think about the future my children, who are in elementary and middle school, will live in, the questions get even bigger. This isn’t to say we’ve found the right answers. It means we’ve begun confronting questions in earnest that are difficult to answer using our existing assumptions.