I sat down with Bianca Lewis of OpenSearch at Open Source Summit India, right after the project’s second OpenSearch Con in the country. Bianca Lewis is the executive director of the OpenSearch Software Foundation.AI brings scale and the ability to do things quickly. We don’t have to work sequentially anymore. But data without context is just data, it’s noise. You need a platform that can recall it and spin up AI agents, you need the data underneath it, and you need the agents to talk to each other to perform tasks but without context, all of that is garbage at scale. OpenSearch has built in some features to address this: short-term agentic memory, so an agent remembers the context of what it was asked over the last week or two, and in 3.7, long-term agentic memory as well, which gives us the context we need to take accurate actions on the data. It’s already generally available.Bianca: Yes, we’ve actually recently announced that. The way we’re doing it is quite unique and exciting for the open source world. We’ve already released the first form of learning content, and in a couple of months we’re going to release an OpenSearch competency course that carries a badge of competency you can put on LinkedIn.
Bianca: In the 3.5 release — we’ve just released 3.7, so we’re two minor versions past that — agentic AI became generally available in OpenSearch, and since then it’s become a central theme of how people run their search and observability stacks. I think we’re ahead of almost any other platform in the world on fully integrated agentic AI.
Table of Contents
Explaining OpenSearch to the average Linux user
This is a bit of an uncomfortable question, more about the Linux Foundation model in general than OpenSearch specifically. MariaDB, Redis, and Elasticsearch all moved from open source to business-source or server-side licenses at some point, largely to protect their business from hyperscalers who weren’t contributing enough but were taking their enterprise customers. Then the Linux Foundation and companies like Amazon fork the original project, as happened with Terraform and OpenTofu, and grow it as the “true” open source version while the original gets left behind. Is there an injustice in that, toward the original projects?Five years in, is OpenSearch still a fork of Elasticsearch, or has it become a product of its own? Does it still pull in any downstream changes from Elastic?Bianca: I think ultimately OpenSearch will be looked at in a couple of layers. OpenSearch is the application level, and the application level sits on top of an infrastructure that supports the architecture of the application.
“The startups of today are the enterprises of tomorrow.”
The ELK Stack brand problem
Bianca: I think it’s a really great question. I don’t ever judge a vendor for deciding to close-source or open-source a project. But without intruding into the particulars of how Elastic or AWS felt at the time, which I can’t speak to, I can share a personal viewpoint.It was a face-to-face conversation and it is lightly edited for clarity. And since it was not a straight forward text interview, the answers are more conversational in nature.Bianca: That is a great question, and the answer, very simply, is yes, it’s an independent project. It’s a completely independent project now that began as a fork, as you correctly said, and in the early days the two were very, very similar. Today the focuses and the strengths have become very different, and today they are completely independent projects. I think the only commonality is that users can still use both platforms for similar kinds of use cases.Bianca: Those are two very valid questions, so let me answer them one by one.
Observability, complexity, and cost
Bianca: There is no doubt that Elastic, Elasticsearch, and Logstash and Kibana, which make up ELK, have been around a long time, and that’s obviously where the fork came from originally. I hope that after five years of OpenSearch developing as an independent and powerful project — 1.7 billion downloads later, 400 individual companies contributing, more than 3,000 contributors, and 15,000 OpenSearch stack users — the differentiation in young developers’ minds is becoming clearer just from the sheer weight of numbers.Today it’s all in a single pipeline into OpenSearch — you see it all together, through different lenses in the same view, and you can automate resolutions. You can use models and AI to interrogate that data, automate resolutions, and use your observability stack for more than troubleshooting — for company-wide SLOs, understanding acceptable error rates, and understanding cost. As you spin up AI agents, OpenSearch is model-agnostic, so you can use any AI model you want. If it’s for a search use case, you can monitor those AI agents with the same platform — monitor the trace of the agent, see which services it’s calling, what the costs are, and make sure it’s doing the right thing. At scale, when you’re running 100,000 queries a minute, we can almost let AI monitor AI for security, safety, and compliance.That’s the beauty of the Linux Foundation model. It’s a nonprofit that nobody owns, and profit isn’t the motive. Yes, there will be cases where hyperscalers fork a project. I think it’s important to understand that once they fork it and give it to a nonprofit like the Linux Foundation, they give up ownership. They don’t own OpenSearch anymore. They could fork it again and keep something in-house, anyone can always do that, but OpenSearch itself can never be closed now, because the Linux Foundation owns it.Today the whole language has changed, because OpenSearch gives you a unified view of the context of your data, which is much more cost-effective than doing log analytics, traces, and metrics separately with tools like Grafana and Prometheus, where everyone looked at the data differently and root-cause analysis took five weeks to put the pieces together. OpenSearch’s 3.7 release, with long-term agentic memory, is already out, and the project’s competency certification track is expected to roll out over the coming months. You can find the project’s documentation, downloads, and community channels at opensearch.org.
It’s FOSS has been helping people use Linux for the past 14 years. Help us stay independent from big tech. Become a Plus member, enjoy ad-free reading and get 5 eBooks.
Observability stacks are getting heavier, more expensive, and more complicated. What is OpenSearch doing about ease of setup and cost?
AI-agent readiness
Maybe if it were named something like “OpenSearch AI” or “OpenAI Search,” it would be more recognizable.Hyperscalers like AWS, Microsoft, Google, IBM, and Oracle make their money from selling compute and infrastructure, not directly from the application. Carrying the burden of growing and innovating an application takes away from their core focus of selling infrastructure, so it makes business sense for them to let the community develop the application while they focus on providing a great service to run it, and put their money where they actually make money. So it’s not just enterprise-focused anymore, it’s for anyone and everyone. Whoever’s requirements fit it, OpenSearch is for them. I don’t think it was ever built to just be for the enterprise. The startups of today are the enterprises of tomorrow.Is OpenSearch now an AI data layer sitting on top of a traditional search stack, or have the two fully merged?
AI data layer vs. traditional search
If a vendor is the owner of an open source project, to me that open-core label is simply a go-to-market strategy to get users in and then sell services, support, and enterprise capability. It’s not what we’d call vendor-neutral. Building a wide community on a vendor-owned open source project is always tricky, and that’s why Mongo, Elastic, Redis, and a few others reached a point where the situation became a threat — the go-to-market had been really successful, they’d built a community, but the biggest customers weren’t converting.Pay once, Enjoy foreverOn top of that, we’re going to do the OpenSearch competency certification proper, and based on that certification, we’re doing accreditations for different vendors and companies. The follow-up to that is specialist certifications. Since OpenSearch is used across quite a few use cases, we’ll have competencies in observability, search, AI applications, and security monitoring. And to renew your certification, instead of retaking the same test every couple of years, as long as you’re advancing to more advanced certifications you keep your original qualifications. We want people to grow and advance within OpenSearch.
On business-source licensing and the Linux Foundation model
The second layer is that the two platforms have grown along different technological paths. Every platform in this space will say “we’re a millisecond faster at this,” or “we support this use case better,” and that’s all true — no platform can honestly say it’s the best at everything. But OpenSearch is really good at a lot of different use cases that unify the data layer, and because it’s the only alternative that isn’t just open core, it’s making young developers distinguish OpenSearch, hopefully much faster than ELK or any other platform.It also helps to look at where each company actually makes its money. Take Elastic. They make money selling Elasticsearch licenses, whether self-hosted or on the cloud, so it makes sense for them to gate advanced features behind a license, because they have to survive as a business and answer to shareholders. What OpenSearch does do is make everything cost-effective. Take observability: in the old days, we had to have it because if something in our infrastructure broke, we had to solve it — we couldn’t afford downtime — but nobody wanted to pay for it because it was seen as a cost center. That’s the subtle reason hyperscalers are so eager to support Linux Foundation projects. It lets them focus on where they make money, and it guarantees these projects stay open, available, and well-maintained rather than becoming orphaned.We covered a lot of ground: whether OpenSearch is still tethered to Elasticsearch in any way, why ELK Stack remains the more recognizable brand among young developers, how agentic AI fits into the platform, and an uncomfortable but necessary question about what happens when vendor-backed open source projects go proprietary.
OpenSearch itself can never be closed now, because the people that own it is the Linux Foundation.
OpenSearch started life in 2021 as a fork of Elasticsearch, born out of Elastic’s move away from an open source license. Five years on, I wanted to know how much of that origin story still holds, and how the project sees itself now that it’s an AI data infrastructure layer with its own roadmap and its own community.Does OpenSearch have plans for a certification track, like Kubernetes has with the CKA?📋I think there are two distinguishing factors. The first is philosophy — how developers want to build their technology stacks conceptually and into the future. Elastic, who owns ELK, even though the license is now back to open source after they closed it, is still owned by Elastic and is open core. That means if a young developer wants to build on Elasticsearch and use the ELK stack, they can use the core version for free, but for any more advanced features or support they’re going to have to pay and be tied into Elastic. Developers who want to build on a platform that’s fully open, where you get the whole thing for free and can choose whether you want support or hosting when you’re ready, have that choice with OpenSearch. It’s a different philosophy.
Certifications
What’s the one thing that most people get wrong about OpenSearch?We’re in the token economy now, so we can add cost information into traces and logs and understand accurately how much each service is really costing us. Observability isn’t just “pay for troubleshooting” anymore — it’s become easier to use and a driver of business imperatives.
The biggest misconception about OpenSearch
ELK Stack is still the more recognizable brand in observability and search. How does OpenSearch compete with that brand recognition among young developers?On complexity: the advantage of proprietary vendors has always been that on day one it’s easy — you buy it, it’s off the shelf, it works, and you get support. With an open platform, you build it on day one, and you get the benefit on day two, three, and four because it’s customized to your needs and you’re not dependent on a vendor. But getting started on day one is a very valid point. Because of that, we’ve recently launched something called Launchpad, which lets you eailylaunch an observability stack, a search application, or any other OpenSearch application, because OpenSearch isn’t just observability, there’s also search, AI applications, and security monitoring. All this takes a few minutes, and we encourage developers to ask the community if they get lost, even though the documentation is all there. I believe the ease of setting up OpenSearch has been addressed incredibly well and quickly.
OpenSearch is the only cloud-native, AI-native platform in the world that’s truly open source and vendor-neutral.
What OpenSearch gives you is that it doesn’t tie you into any one vendor’s choices. You can have that AI infrastructure layer but choose your own infrastructure. You can self-host, or go to one of the hosted services, like our OpenSearch service on NetApp, Oracle, and AWS and choose whatever fits your model. In terms of features and how things run with agentic AI, we don’t really say this works better or worse than that. We say everything works and works really well, and with OpenSearch you retain control over how you want to build it, without being tied into any one approach.Bianca: There are so many things people get wrong about OpenSearch, but I think the biggest one is not recognizing that, as an AI data infrastructure layer, OpenSearch is the only cloud-native, AI-native platform in the world that’s truly open source and vendor-neutral. I don’t know if people really understand that.
Ultimately it takes time for a project’s original founder to say “now we can let the community manage it.” What that actually means is that over the years, instead of maintaining the application with a full-time in-house engineering team, the founder becomes one of the companies contributing to a shared engineering effort. You get all the benefit, but the innovation and cost become shared, and the community retains the choices, which is always good for business.
I find it interesting that Elastic changed back to an open source license after closing it — because they were losing the community, but they’d closed it because they were losing enterprise customers. They were fine with small players using the free product but wanted the big-money customers. The challenge is that often you want your cake and to eat it too. Once you build your brand on community involvement and then tell the community you’re closing it, even if you reopen it later, a lot of the community will just say “thanks, but no thanks.”
