| ESP Journal of Engineering & Technology Advancements |
| © 2026 by ESP JETA |
| Volume 6 Issue 4 |
| Year of Publication : 2026 |
| Author : Mohit Mittal |
10.56472/25832646/JETA-V6I4P105 |
Mohit Mittal, 2026. AI-Native Data Architectures for Agentic Applications: A Comparative Evaluation of Lakehouse, Warehouse, and OLAP Technologies, Volume 6 Issue 4: 46-56.
Large language model (LLM) agents are emerging as a new class of client for analytical data systems. Unlike a human analyst or a dashboard, an agent explores a database by issuing bursts of heterogeneous, partially redundant queries, requests individual entities during a reasoning loop, and expects the data it reads to reflect writes that occurred seconds earlier. Recent database research has termed this behavior agentic speculation and argued that data systems should be redesigned around it. Practitioners, meanwhile, must choose today among three mature architecture families: the cloud data warehouse, the open- table- format lakehouse, and the real- time OLAP engine. This manuscript compares the three for agentic workloads. We first derive seven data- layer requirements for agentic applications from the literature on agent behavior, cloud analytics workloads, and text- to- SQL benchmarks. We then define a reproducible agentic query workload with four parts: schema and sample probes, a speculative analytic burst with 57% exact- repeat queries, entity point lookups, and a micro- batch freshness test. We run it on single- node, open- source proxies for each family: DuckDB native storage (warehouse- style), Delta Lake tables scanned by DuckDB (lakehouse), and ClickHouse MergeTree through chDB (real- time OLAP), all on 5 million order events. The warehouse- style proxy had the lowest median latency on the speculative burst (7. 7.2 ms against 21. 3 ms and 24. 3 ms). The OLAP proxy was 15 times faster than the lakehouse proxy on entity lookups (2. 2.8 ms against 42. 1 ms). The lakehouse proxy made micro- batch writes visible roughly seven to ten times more slowly than the other two (50. 3 ms median, against 5. 5.2 ms and 7. 7.3 ms), and it accumulated 100 new data files after 100 appends. Per- query snapshot resolution accounted for about 17% of the lakehouse' s burst latency. No single architecture met every requirement. We therefore propose an AI- native access layer that sits above any of the three and supplies semantic metadata, freshness watermarks, result caching, and cost guardrails. We also provide a requirement- to- architecture selection matrix. The results are small- scale and proxy- based. We report them as directional evidence with a released methodology, not as vendor rankings.
[1] Dageville, B., Cruanes, T., et al. (2016). The Snowflake elastic data warehouse. In Proceedings of the ACM SIGMOD International Conference on Management of Data (pp. 215–226).
[2] Vuppalapati, M., Miron, J., et al. (2020). Building an elastic query engine on disaggregated storage. In Proceedings of the 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI).
[3] Melnik, S., Gubarev, A., et al. (2010). Dremel: Interactive analysis of web-scale datasets. Proceedings of the VLDB Endowment, 3(1–2), 330–339.
[4] Armbrust, M., Ghodsi, A., et al. (2021). Lakehouse: A new generation of open platforms that unify data warehousing and advanced analytics. In Proceedings of the Conference on Innovative Data Systems Research (CIDR).
[5] Armbrust, M., Das, T., et al. (2020). Delta Lake: High-performance ACID table storage over cloud object stores. Proceedings of the VLDB Endowment, 13(12), 3411–3424.
[6] Behm, A., Palkar, S., et al. (2022). Photon: A fast query engine for lakehouse systems. In Proceedings of the ACM SIGMOD International Conference on Management of Data.
[7] Schulze, R., Schreiber, T., et al. (2024). ClickHouse, lightning fast analytics for everyone. Proceedings of the VLDB Endowment, 17(12), 3731–3744.
[8] Yang, F., Tschetter, E., et al. (2014). Druid: A real-time analytical data store. In Proceedings of the ACM SIGMOD International Conference on Management of Data (pp. 157–168).
[9] Im, J.-F., Gopalakrishna, K., et al. (2018). Pinot: Realtime OLAP for 530 million users. In Proceedings of the ACM SIGMOD International Conference on Management of Data.
[10] Yao, S., Zhao, J., et al. (2023). ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR).
[11] Anthropic. (2024). Introducing the Model Context Protocol.
[12] Liu, S., Ponnapalli, S., et al. (2026). Supporting our AI overlords: Redesigning data systems to be agent-first. In Proceedings of the Conference on Innovative Data Systems Research (CIDR).
[13] Lei, F., Chen, J., et al. (2025). Spider 2.0: Evaluating language models on real-world enterprise text-to-SQL workflows. In International Conference on Learning Representations (ICLR).
[14] Li, J., Hui, B., et al. (2023). Can LLM already serve as a database interface? A big bench for large-scale database grounded text-to-SQLs. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track.
[15] Tan, J., Ghanem, T., et al. (2019). Choosing a cloud DBMS: Architectures and tradeoffs. Proceedings of the VLDB Endowment, 12(12).
[16] van Renen, A., & Leis, V. (2023). Cloud analytics benchmark. Proceedings of the VLDB Endowment, 16(6), 1413–1425.
[17] Jain, P., Kraft, P., et al. (2023). Analyzing and comparing lakehouse storage systems. In Proceedings of the Conference on Innovative Data Systems Research (CIDR).
[18] Camacho-Rodríguez, J., Agrawal, A., et al. (2024). LST-Bench: Benchmarking log-structured tables in the cloud. Proceedings of the ACM on Management of Data, 2(1).
[19] van Renen, A., Horn, D., et al. (2024). Why TPC is not enough: An analysis of the Amazon Redshift fleet. Proceedings of the VLDB Endowment, 17(11), 3694–3706.
[20] Agiwal, A., Khandelwal, K., et al. (2021). Napa: Powering scalable data warehousing with robust query performance at Google. Proceedings of the VLDB Endowment, 14(12).
[21] Blue, R., Weeks, D., & Narayanan, A. (2025). Apache Iceberg™ v3: Moving the ecosystem towards unification. Databricks Blog.
[22] DuckDB Team. (2026). DuckLake v1.0: The lakehouse format built on SQL reaches production-readiness.
[23] Balaji, R. (2025) ‘AgenticSecOps for Microservice APIs: Policy-Constrained Multi-Agent Investigation and Response to Application-Layer Incidents’, ESP Journal of Engineering & Technology Advancements, 5(3), pp. 212-221. doi: 10.56472/25832646/JETA-V5I3P127
[24] Abadi, D., Boncz, P., et al. (2013). The design and implementation of modern column-oriented database systems. Foundations and Trends in Databases, 5(3), 197–280.
[25] Castro Fernandez, R., Elmore, A. J., et al. (2023). How large language models will disrupt data management. Proceedings of the VLDB Endowment, 16(11).
[26] Gao, D., Wang, H., et al. (2024). Text-to-SQL empowered by large language models: A benchmark evaluation. Proceedings of the VLDB Endowment, 17(5).
[27] Pan, J. J., Wang, J., & Li, G. (2024). A survey of vector database management systems. The VLDB Journal, 33.
[28] Malkov, Y. A., & Yashunin, D. A. (2020). Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(4), 824–836.
[29] Raasveldt, M., & Mühleisen, H. (2019). DuckDB: An embeddable analytical database. In Proceedings of the ACM SIGMOD International Conference on Management of Data (pp. 1981–1984).
Agentic AI, Data Architecture, Lakehouse, Data Warehouse, Real-Time OLAP, Delta Lake, Apache Iceberg, Clickhouse, Duckdb, Text-To-SQL, Model Context Protocol, Benchmarking.