Everyone Has Access to a Personal Jarvis: Kartik Paramasivam on Agentic Engineering, GPU Procurement, and What Actually Breaks at Scale

Q1.  Kartik, you have built massive data infrastructures at both LinkedIn and Pinterest. As we transition from traditional software engineering to “Agentic Engineering,” how do the core competencies of a top-tier engineer change when writing precise specifications and prompts becomes more critical than simply “vibing with code”?

The answer to this question has been evolving as the frontier models are evolving and improving every day. Given the changing foundations, a key expectation of a top tier engineer is to adapt their software engineering life cycle to best leverage the strengths of the models and the harnesses. Ensuring that the coding agents are fed with the right context and goals while creating appropriate guardrails are critical. At the end the goal is to build the best product/system that achieves the desired business outcomes.

Towards this goal, several of the key competencies of top tier engineers remain the same as before. Engineers need to be deeply rooted in understanding the requirements (the what) and business problems (the why) that they have set out to solve. Note that by ‘requirements’ I mean both the product requirements and non-functional requirements like latency, scale needs, security, compliance etc. The importance of evaluations and testing that validate that the system is meeting the desired behavior and outcomes has become even more important.  

Top engineers still have to fully understand the architecture and design, understand the fundamentals of how the systems are working, and can zoom in and understand any level of detail of the system when needed. They apply their unique knowledge of their domain, their company and the industry as they steer the coding agents.

Amongst the biggest things to call out is that top engineers are adopting and implementing best practices for agentic engineering for their systems and teams.They are not just making themselves more productive; but instead, the best practices that they are establishing are making every engineer on their team productive and also improving the quality of their systems.

In terms of what is changing, the biggest change is that natural language is becoming the language to create software as opposed to Java, C++, Rust et. all.  Deep experience in domain specific frameworks is not that important now. Human code review is still an activity performed by engineers in many projects and companies. However, in many cases the human engineer is just ensuring that a code review agent has reviewed the change and tries to look at the pull request to ensure the right things are happening (e.g., the change to intent is appropriate,  appropriate tests are being updated/checked in along with the PR, etc.). 

Q2. AI code generation is driving development velocity through the roof. However, in an engineering organization the size of Pinterest, where do the new bottlenecks emerge once writing code is no longer the slow part? Furthermore, how do you effectively manage and allocate “token budgets” across a massive developer team without stifling innovation?

As we solve existing bottlenecks, new bottlenecks emerge. Writing code is not a bottleneck anymore, but code reviews still are. As companies move to automated code reviews for low risk changes, the bottleneck moves again.  For e.g. if you are an online consumer company and have a heavy culture of experimentation, the bottleneck moves to how many parallel online experiments can you run given the scale of your traffic. To speed up experimentation, the question also is whether you are good at running experiments offline. Once you can run experiments offline, then you want to automate the entire training loop.  For example, you want to be able to create an auto-research loop that allows agents to run several 100s of experiments to create, update and tune machine learning models and their features.

Our job as engineers right now is to keep at it and find and fix the next bottleneck.

Eventually, the bottleneck that will remain in most companies is human to human communication and processes involving humans. Human engineers and product managers and designers still have to discuss and align on the shape of the product and the business outcomes. Companies that are able to do this effectively and efficiently will flourish.

On token budgets, most companies are building custom systems to manage token budgets across all the different model providers (most companies use multiple model providers). Such a system typically optimizes for the following objectives. (1) Provide flexibility to support the top projects (2) Adapt limits for everyone as the spend grows (3) Give organizational leaders the flexibility to move token budgets between projects.    

In addition to token budget management, there obviously needs to be a lot of focus on ensuring engineers are using tokens efficiently and focus on high ROI activities.

Q3.  Procuring GPU capacity today feels like a complex logistical operation. What is your engineering framework for deciding whether to keep GPUs within a single cloud provider versus leveraging multiple niche GPU cloud vendors—especially when factoring in data latency, networking overhead, and the physical location of your clusters?

This is honestly a tough problem.   

The main constraints are that (a) GPUs are often not available in your primary region/s (b) Cost of data transfer across regions and across clouds is very high and adds up.  For example, depending on whether your data lake is petabyte scale or exabyte scale changes what is practical for your company. (c ) The financial envelope that you have to operate under.   

The main objective here is to create flexibility for the business.You have to create flexible data pipelines that can push the minimum amount of data required to the right locations back and forth for your training runs. Embracing an open source tech stack makes it much easier to perform training and inference across providers. Most companies are also adopting open source /open weight LLMs, which are easier to fine tune and change. And now increasingly it is becoming easier to run these models on specialized inference hardware (e.g. Trainium)  

Q4. You were on the front lines at LinkedIn when Apache Kafka was being scaled to handle over a trillion messages a day, and you heavily championed frameworks like Apache Samza for stateful stream processing. Back then, the challenge was building real-time data highways for analytics.

Fast forward to today: as we move into generative AI, the underlying infrastructure must handle entirely new data modalities like massive vector embeddings and dynamic prompt contexts. How is the foundational “pub-sub and streaming” architecture you helped pioneer evolving to sustain real-time AI, and what are the limits of traditional big-data open-source frameworks when confronted with modern LLM pipelines?

Yes, I had the opportunity to build and influence foundational pub-sub and data processing technologies first in Azure and then at LinkedIn.  After all these years, PubSub and Stream Processing continue to be critical for real time AI.   Pinterest users have a lot of varied interests and explore different interests at different times.   We need Pinterest recommendations to respond quickly based on what the user is looking for in their current session.  This means that we need to process new activity within seconds and ensure the features/signals that capture this new activity are available for the retrieval and ranking systems that put together the next set of recommendations for the user.

Although Flink SQL and similar initiatives have made it easier to build real time pipelines the truth is that building end to end real time data pipelines continues to have higher engineering and operational cost than building offline batch pipelines where the guarantees are much weaker and tend to be more forgiving of failures and delays.   

But engineering teams that push through the initial cost and implement near real time data processing are able to create very interactive and delightful customer experiences. 

Q5. When looking at long-form agentic work—where autonomous agents operate independently over days on large enterprise code repositories—how much system access do you safely grant them? What architectural guardrails do you put in place to manage the inherent risks of autonomous productivity agents?

This is a billion dollar question and again, an evolving situation.   A few months ago, the big concern with agent access was to protect against the lethal trifecta.  But now the concern is not just about prompt injection and the risk of exfiltration of company data.  After the Hugging Face hack, the concerns have expanded. It is easy to give a prompt to a misaligned AI running inside your company that can wreak havoc both inside and potentially outside the company. Also, a bad actor can easily use an open source frontier model to exploit zero day vulnerabilities in services and get access to critical systems.

Unfortunately, there isn’t a silver bullet here. Everyone has to get back to the basics and address security gaps in their own companies and follow best practices.   

The top ones that come to mind are (a) Find vulnerabilities using the frontier models and patch them aggressively  (b) Adhere to the rule of two  (c )  Ensure all internal endpoints have adequate access control and audit the ACLs (d )  Give agents narrow access whenever you can and ensure that access is scoped to the user on behalf of which the agent is processing the data. (e) Ensure critical infra systems have dual access controls so that a misconfigured agent can’t make critical changes autonomously without a human in the loop. (f) Every company needs to become great at observability e.g., intrusion detection, monitoring of all egress out of critical networks, and closely monitoring what the agents are doing.

---

Q6.  You have steered the technological scaling of two of the world’s most influential platforms. Looking back at your journey from LinkedIn to Pinterest, what is one major architectural bet or engineering intuition you had that went completely against the grain at the time, but ultimately proved to be one of your greatest successes?

There are probably a few success stories and many failures too.   Maybe what might be interesting is to discuss the lessons learned as opposed to details of one specific example.  Here are some of my top lessons.

1. Watch what you depend on: Sometimes you take a dependency on a cloud based service to achieve faster go to market. But you don’t always know the reliability and performance characteristics of this service for your workload and at your scale.  Most new services in the cloud take many, many years to mature. So beware of the guarantees, or lack of it, you may be getting from the dependencies you take.   
2. To use an open source system or to build in-house: Every open source community is different. Some of them are very tightly controlled by one company. If you take a dependency on such a system and it is not a perfect fit and you need many changes, then you have to be prepared to fork the codebase. Now that we have the power of AI, it can be easier to build smaller systems from scratch ourselves, instead of forking an existing open source system. For complicated systems like databases, forking and keeping the fork updated is still the way to go.  
3. Beware the Black Swan Events: Given where we are in the world and AI, the rate of occurrence of black swan events is accelerating.This typically means that the foundational assumptions that you might have made in your business or technology can get completely inverted on their head. For example, several such events have had a huge impact on the available compute capacity e.g., Covid outbreak, Russia-Ukraine war, The Chat GPT moment, Exponential Rise of OpenAI and Anthropic, Changes in legislation. So if you are going to start long term projects, be prepared to push through the Black Swan Events.

Qx. Anything else you wish to add?

It is a very exciting time to be in tech. With AI now, everyone has access to a personal Jarvis.  But to take advantage of it, you still need a builder’s mindset. You still need curiosity and the interest to tinker. There probably has never been a better time to be a builder.

……………………..………………………………………………

Kartik Paramasivam

Kartik is Pinterests’  Chief Architect, where he leads the direction of Pinterest’s tech stack, foundational platforms and infrastructure and supports the overall growth of Pinterest’s top level goals for engagement and revenue.      Kartik brings deep expertise in building internet-scale distributed systems and data platforms that power products used by hundreds of millions of people.   Before joining Pinterest in 2024,  Kartik spent 10 years at LinkedIn, most recently serving as Vice President Engineering for Data Infrastructure.  Earlier there, he was a Distinguished engineer and led LinkedIn’s migration to Microsoft Azure.    Prior to LinkedIn,  he spent close to 15 years at Microsoft, where he worked on the 1st and 2nd gen service offerings (ServiceBus/EventHubs) as part of Microsoft Azure.  Kartik has contributed to 50+ patents and several research papers and open source systems.