On Neurelo Database Abstraction Platform. Q&A with Chirag Shah

Q1. You have founded Neurelo. What is it?

Neurelo is an API-first database abstraction platform that provides a comprehensive data access layer designed to streamline database programming and management for developers. Neurelo instantly transforms databases into REST or GraphQL APIs, creating an abstraction layer that allows developers to interact with databases through APIs rather than direct database programming. 

This approach enables a modern architecture, streamlines the development process and makes it way easier to integrate databases into applications. The platform supports major database systems including: PostgreSQL, MongoDB, and MySQL. Neurelo’s platform can be consumed as a SaaS or self-hosted in customer environments.

Q2. You claim that database abstractions are long overdue in software development. Why?

As developers build an application, they begin by focusing on data modeling to align application requirements with data definitions and schema design, which involves integrating database drivers and using the database’s query language. As new features are added, the schema requires reassessment and adjustments, leading to migrations across all environments, including development, testing, QA, and production.

Releasing to production creates additional dependencies for developers, including ongoing optimizations for performance as the application scales, adjusting resources for traffic peaks, and managing data access privacy for various user classes. These tasks require access to skilled and limited technical expertise.

As the application evolves, new requirements arise, including the addition of caching layers, data distribution across geographic regions, and features like full-text keyword search to enhance user engagement. Additionally, various teams within the organization, including data teams for model training and analytics, seek access to application data for different use cases.

Engineering teams commonly face these challenges, leading to the use of numerous disparate tools for specific tasks, which creates complexity that slows innovation, raises project costs and risks, and contributes to team attrition.

Database abstraction provides flexibility, reduces the risk of vendor lock-in, and simplifies code maintenance, especially when scaling or evolving applications. It also makes it easier to integrate new databases or features while minimizing disruption to existing systems. Current abstractions like ORM frameworks fall short, adding technical debt and limiting flexibility.

Some organizations have addressed these challenges by creating custom abstraction layers between their applications and databases, which can be costly to build and maintain, diverting engineering resources from innovation. 

To enhance developer productivity and reduce costs, it’s essential to converge data tools and abstract dependencies across the entire software development lifecycle, as managing data definitions, integration, programming, monitoring, scaling, securing, and usage are interconnected workflows.

Q3. Why do you believe that the dependencies between applications and databases are leading to inefficiencies and high costs? Can you give us an example?

Working with data presents several complexities for developers in application development, particularly in the areas of velocity, scaling and security.

1. App development velocity
   1. Data Modeling: Designing an effective data model that accurately reflects the business logic can be intricate, which often leads to challenges during schema migration in relational databases when altering existing structures without disrupting ongoing operations. Additionally, maintaining schema governance in both relational and NoSQL databases requires ensuring consistency, version control, and compliance with evolving application requirements, complicating collaborative development efforts.
   2. Query languages: Pretty much each database has its own query language, which requires developers to learn and adapt to different syntaxes and features for each database system. This can lead to increased complexity in writing and maintaining queries, as well as difficulties in optimizing performance across multiple database platforms. Even within SQL, pretty much all relations databases such as Oracle, MySQL, Postgres have their own SQL dialect
   3. CI/CD integration: Developers face challenges in integrating databases into their CI/CD workflows due to the need for consistent schema migrations and data seeding across different environments, which can lead to discrepancies and errors if not managed properly. Additionally, automating database tests and ensuring that changes are backward-compatible can complicate the deployment process, potentially resulting in downtime or data loss.
2. Scaling challenges
   1. Traffic Spikes: During peak usage and/or unpredictable traffic surges, databases may struggle with a sudden influx of read/write operations and can overwhelm resources. Solutions like read replicas, load balancing and auto scaling are essential as they help distribute the load but require careful handling. Replication can distribute reads, but write-heavy loads require horizontal scaling and/or complex partitioning strategies.
   2. Connection Storms: In a large environment when many clients attempt to connect to the database simultaneously, it has a high risk of connection storms issue, overwhelming the database with too many open connections. This can lead to bottlenecks and failures. Connection pooling helps manage this by limiting the number of open connections, but careful consideration of limits and timeouts is needed.
   3. Sharding and Partitioning: Horizontal scaling (adding more servers) is complex with relational databases due to maintaining data consistency across distributed nodes. Partitioning data across multiple servers (sharding) is crucial for performance but increases complexity in managing distributed transactions, ensuring query efficiency, and maintaining consistency across shards. NoSQL databases can scale horizontally more easily, but they often sacrifice ACID properties, requiring developers to handle consistency issues manually.
3. Security complexities
   1. Fine-Grained Access Control: Granular control is required for restricting access at the table, row, or column level. For example, in an ecommerce system, a user may have access to customer order (tables) but may be restricted from viewing sensitive fields like credit card information (columns) or certain customers based on geography.  Column-level encryption adds another layer of security for sensitive information like credit card numbers. Databases like PostgreSQL and Oracle offer row-level security (RLS) and column-level encryption to meet these needs.
   2. Read/Write Separation: Assigning permissions based on the operation (read vs. write access) helps to minimize exposure. Users might be able to read large datasets but may have limited write access to prevent unauthorized updates or deletions.
   3. Audit & Monitoring: Read and write level logging for access attempts, query patterns, and changes to sensitive data could help in detecting and preventing unauthorized activities.

Q4. What is the Neurelo approach instead?

For years, we faced significant challenges in building, scaling, and securing applications, often attributing our bottlenecks to databases. However, we discovered that while both modern and legacy databases are robust and reliable, the real issue lies in the interaction layer between applications and databases. This realization inspired the creation of Neurelo. We automatically transform any supported database into APIs, leveraging AI to simplify, automate, and streamline complex data operations into intuitive processes.

Neurelo auto-generates database schemas and creates REST and GraphQL APIs, equipping developers with features like schema versioning, AI-powered mock data generation, and Git integration. 

While we automate and simplify many aspects of app development, at runtime, Neurelo functions as both an API server and a database gateway (proxy). 

This approach unlocks endless possibilities for addressing scaling, security, and other requirements without the need for a complex setup of custom or off-the-shelf tools bridging the application and database. For example, you can easily define access policies to restrict “all writes” to specific tables or columns, and we operationalize these changes with virtually no modifications required on either the application or database side. Additionally, you can implement read-level load balancing to distribute read traffic across secondary nodes, safeguarding the primary node from slowdowns or outages.

Another example is when developers decide to introduce caching solutions like Redis into their architecture. Typically, this requires significant changes to the application code and various other adjustments, which can be cumbersome. However, with Neurelo, you can simply select the option to redirect “read traffic” to the cache layer, and that’s it. We can operationalize caching within any application architecture in just days instead of months.

Our approach ensures that the application and database stack is sufficiently decoupled to scale independently while remaining performant enough to meet demanding application performance requirements. Neurelo acts as a natural abstraction layer that enhances scalability, serving as a shock-absorber for both predictable and unpredictable growth while accommodating more complex requirements, such as fine-grained access control.

Q5. With Neurelo you claim to be able to simplify database interaction. How?

Neurelo simplifies database interaction through several key features:

• Instant API Generation: Neurelo automatically generates schema-aware REST and GraphQL APIs for all database operations, including CRUD operations and advanced queries like joins and aggregations.
• AI-Powered Assistance: The platform includes an AI assistant trained on database syntax and semantics to help developers formulate complex queries, which can then be instantly deployed as custom API endpoints.
• GitSchema: Neurelo offers a schema-as-code feature called GitSchema, which is particularly useful for teams managing complex, evolving database structure.  The platform provides an AI-Powered schema builder for quick schema generation & modification, an ER view for understanding data relationships, schema codification in JSON and YAML formats, and integration with development workflows for version control and schema migration.
• Observability & Performance Monitoring: Built-in tools provide real-time insights into performance, query efficiency, and API usage, giving developers a clear understanding of how their applications interact with the database.
• Support for Multiple CI/CD Environments: Neurelo allows developers to seamlessly build, test, and deploy their schema changes across multiple environments such as dev, test, staging and production across their SDLC.
• Synthetic Data Generation: Neurelo can generate high-fidelity mock data, useful for testing and development without the need for sensitive production data.
• Flexible Integration: Neurelo’s REST and GraphQL APIs can be used from just about any programming language and the platform also supports SDKs for multiple languages, including Python, Go, and TypeScript, making it adaptable to various tech stacks.

Neurelo’s API-first approach provides developers with the tools to reduce the complexity of backend development, offering faster development times, reduced overhead, and better scalability. By turning databases into APIs, it eliminates the need for developers to work directly with raw database code, thus streamlining application development.  

…………………………………………….

Chirag Shah is the co-founder and CEO of Neurelo, a Cloud based API-first database abstraction platform for developers. He brings over 20 years of experience in enterprise software, spanning data infrastructure, cloud platforms, and application management. Previously, he served as Vice President of the Product organization at MongoDB, overseeing core database products. Before MongoDB, Chirag co-founded Zero Systems and held product and GTM leadership roles at VMware, HP Software, EDS, and Loudcloud.

Sponsored by Neurelo.