> Safe languages insert additional machine branches to do things like verify that array accesses are in-bounds. In correct code, those branches are never taken. That means that the machine code cannot be 100% branch tested, which is an important component of SQLite's quality strategy.
Huh it's not everyday that I hear a genuinely new argument. Thanks for sharing.
So is the argument that safe langs produce stuff like:
// pseudocode
if (i >= array_length) panic("index out of bounds")
that are never actually run if the code is correct? But (if I understand correctly) these are checks implicitly added by the compiler. So the objection amounts to questioning the correctness of this auto-generated code, and is predicated upon mistrusting the correctness of the compiler? But presumably the Rust compiler itself would have thorough tests that these kinds of checks work?
Someone please correct me if I'm misunderstanding the argument.
It's the sort of argument that I wouldn't accept from most people and most projects, but from Dr Hipp isn't most people and Sqlite isn't most projects.
I wonder if this problem could be mitigated by not requiring coverage of branches that unconditionally lead to panics. or if there could be some kind of marking on those branches that indicate that they should never occur in correct code
I think those branches are often not there because it's provably never going out of bounds. There are ways to ensure the compiler knows the bounds cannot be broken.
It has async I/O support on Linux with io_uring, vector support, BEGIN CONCURRENT for improved write throughput using multi-version concurrency control (MVCC),
Encryption at rest, incremental computation using DBSP for incremental view maintenance and query subscriptions.
Time will tell, but this may well be the future of SQLite.
The moment turso becomes stable , SQLite will inevitably fade away with time if they don’t rethink how contributions should be taken. I honestly believe the Linux philosophy of software development will be what catapults turso forward.
> All that said, it is possible that SQLite might one day be recoded in Rust. Recoding SQLite in Go is unlikely since Go hates assert(). But Rust is a possibility. Some preconditions that must occur before SQLite is recoded in Rust include:
- Rust needs to mature a little more, stop changing so fast, and move further toward being old and boring.
- Rust needs to demonstrate that it can be used to create general-purpose libraries that are callable from all other programming languages.
- Rust needs to demonstrate that it can produce object code that works on obscure embedded devices, including devices that lack an operating system.
- Rust needs to pick up the necessary tooling that enables one to do 100% branch coverage testing of the compiled binaries.
- Rust needs a mechanism to recover gracefully from OOM errors.
- Rust needs to demonstrate that it can do the kinds of work that C does in SQLite without a significant speed penalty.
1. Rust has had ten years since 1.0. It changes in backward compatible ways. For some people, they want no changes at all, so it’s important to nail down which sense is meant.
2. This has been demonstrated.
3. This one hinges on your definition of “obscure,” but the “without an operating system” bit is unambiguously demonstrated.
4. I am not an expert here, but given that you’re testing binaries, I’m not sure what is Rust specific. I know the Ferrocene folks have done some of this work, but I don’t know the current state of things.
5. Rust as a language does no allocation. This OOM behavior is the standard library, of which you’re not using in these embedded cases anyway. There, you’re free to do whatever you’d like, as it’s all just library code.
6. This also hinges on a lot of definitions, so it could be argued either way.
“None of the safe programming languages existed for the first 10 years of SQLite's existence. SQLite could be recoded in Go or Rust, but doing so would probably introduce far more bugs than would be fixed, and it may also result in slower code.”
Modern languages might do more than C to prevent programmers from writing buggy code, but if you already have bug-free code due to massive time, attention, and testing, and the rate of change is low (or zero), it doesn’t really matter what the language is. SQLIte could be assembly language for all it would matter.
So, the argument for keeping SQLite written in C is that it gives the user the choice to either:
- Build SQLite with Yolo-C, in which case you get excellent performance and lots of tooling. And it's boring in the way that SQLite devs like. But it's not "safe" in the sense of memory safe languages.
- Build SQLite with Fil-C, in which case you get worse (but still quite good) performance and memory safety that exceeds what you'd get with a Rust/Go/Java/whatever rewrite.
Recompiling with Fil-C is safer than a rewrite into other memory safe languages because Fil-C is safe through all dependencies, including the syscall layer. Like, making a syscall in Rust means writing some unsafe code where you could screw up buffer sizes or whatnot, while making a syscall in Fil-C means going through the Fil-C runtime.
The point about bounds checking in `safe' languages is well taken, it does prevent 100% test coverage. As we all agree, SQLite has been exhaustively tested, and arguments for bounds checking in it are therefore weakened. Still, that's not an argument for replicating this practice elsewhere, not unless you are Dr Hipp and willing to work very hard at testing. C.A.R. Hoare's comment on eliminating runtime checks in release builds is well-taken here: “What would we think of a sailing enthusiast who wears his life-jacket when training on dry land but takes it off as soon as he goes to sea?”
I am not Dr Hipp, and therefore I like run-time checks.
One thing I found especially interesting is the section at the end about why Rust isn’t used. It leaves open the door and at least is constructive feedback to the Rust community
The fact that a C library can easily be wrapped by just about any language is really useful. We're considering writing a library for generating a UUID (that contains a key and value) for reasons that make sense to us and I proposed writing this in C so we could simply wrap it as a library for all of the languages we use internally rather than having to re-implement it several times. Not sure if we'll actually build this library but if we do it will be in C (I did managed to get the "wrap it for each language" proposal pre-approved).
You can expose a C interface from many languages (C++, Rust, C# to name a few that I've personally used). Instead of introducing a new language entirely, it's probably better to write the library in one of the languages you already use.
It is. You can also write it in C++ or Rust and expose a C API+ABI, and then you're distributing a binary library that the OS sees as very similar to a C library.
Occasionally when working in Lua I'd write something low-level in C++, wrap it in C, and then call the C wrapper from Lua. It's extra boilerplate but damn is it nice to have a REPL for your C++ code.
Edit: Because someone else will say it - Rust binary artifacts _are_ kinda big by default. You can compile libstd from scratch on nightly (it's a couple flags) or you can amortize the cost by packing more functions into the same binary, but it is gonna have more fixed overhead than C or C++.
Safe languages insert additional machine branches to do things like verify that array accesses are in-bounds. In correct code, those branches are never taken. That means that the machine code cannot be 100% branch tested, which is an important component of SQLite's quality strategy.
Rust needs to mature a little more, stop changing so fast, and move further toward being old and boring.
Rust needs to demonstrate that it can do the kinds of work that C does in SQLite without a significant speed penalty.
If the branch is never taken, and the optimizer can prove it, it will remove the check. Sometimes if it can’t actually prove it there’s ways to help it understand, or, in the almost extreme case, you do what I commented below.
Yeah I don't understand the argument. If you can't convince the compiler that that branch will never be taken, then I strongly suspect that it may be taken.
> Safe languages insert additional machine branches to do things like verify that array accesses are in-bounds. In correct code, those branches are never taken. That means that the machine code cannot be 100% branch tested, which is an important component of SQLite's quality strategy.
This is annoying in Rust. To me array accesses aren't the most annoying, it's match{} branches that will never been invoked.
There is unreachable!() for such situations, and you would hope that:
if array_access_out_of_bounds { unreachable!(); }
is recognised by the Rust tooling and just ignored. That's effectively the same as SQLite is doing now by not doing the check. But it isn't ignored by the tooling: unreachable!() is reported as a missed line. Then there is the test code coverage including the standard output by default, and you have to use regex's on path names to remove it.
But if you don't have the bounds checks in machine code, then you don't have bounds checks.
I suppose SQLite might use a C linter tool that can prove the bounds checks happen at a higher layer, and then elide redundant ones in lower layers, but... C compilers won't do that by default, they'll just write memory-unsafe machine code. Right?
> Rust needs to mature a little more, stop changing so fast, and move further toward being old and boring.
Talking about C99, or C++11, and then “oh you need the nightly build of rust” were juxtaposed in such a way that I never felt comfortable banging out “yum install rust” and giving it a go.
Other than some operating systems projects, I haven’t run into a “requires nightly” in the wild for years. Most users use the stable releases.
(There are some decent reasons to use the nightly toolchain in development even if you don’t rely on any unfinished features in your codebase, but that means they build on stable anyway just fine if you prefer.)
I think zig generally composes better than rust. With rust you pretty much have to start over if you want reusable / composable code, that is not use the default std. Rust has small crates for every little thing because it doesn't compose well, as well to improve compile times. libc in the default std also is major L.
> Nearly all systems have the ability to call libraries written in C. This is not true of other implementation languages.
From section "1.2 Compatibility". How easy is it to embed a library written in Zig in, say, a small embedded system where you may not be using Zig for the rest of the work?
Also, since you're the submitter, why did you change the title? It's just "Why is SQLite Coded in C", you added the "and not Rust" part.
The article allocates the last section to explaining why Rust is not a good fit (yet) so I wanted the title to cover that part of the conversation since I believe it is meaningful. It illustrates the tradeoffs in software engineering.
> Safe languages insert additional machine branches to do things like verify that array accesses are in-bounds. In correct code, those branches are never taken. That means that the machine code cannot be 100% branch tested, which is an important component of SQLite's quality strategy.
Huh it's not everyday that I hear a genuinely new argument. Thanks for sharing.
So is the argument that safe langs produce stuff like:
that are never actually run if the code is correct? But (if I understand correctly) these are checks implicitly added by the compiler. So the objection amounts to questioning the correctness of this auto-generated code, and is predicated upon mistrusting the correctness of the compiler? But presumably the Rust compiler itself would have thorough tests that these kinds of checks work?Someone please correct me if I'm misunderstanding the argument.
It's the sort of argument that I wouldn't accept from most people and most projects, but from Dr Hipp isn't most people and Sqlite isn't most projects.
I wonder if this problem could be mitigated by not requiring coverage of branches that unconditionally lead to panics. or if there could be some kind of marking on those branches that indicate that they should never occur in correct code
I think those branches are often not there because it's provably never going out of bounds. There are ways to ensure the compiler knows the bounds cannot be broken.
This is ignoring the elephant in the room: SQLite is being rewritten in Rust and it's going quite well. https://github.com/tursodatabase/turso
It has async I/O support on Linux with io_uring, vector support, BEGIN CONCURRENT for improved write throughput using multi-version concurrency control (MVCC), Encryption at rest, incremental computation using DBSP for incremental view maintenance and query subscriptions.
Time will tell, but this may well be the future of SQLite.
The moment turso becomes stable , SQLite will inevitably fade away with time if they don’t rethink how contributions should be taken. I honestly believe the Linux philosophy of software development will be what catapults turso forward.
In the link you provided, this is what I read: "An in-process SQL database, compatible with SQLite."
Compatible with SQLite. So it's another database?
Yeah, I don't think it even counts as a fork - it's a ground-up re-implementation which is already adding features that go beyond the original.
so its sqlite++ since they added bunch of things on top of that
> All that said, it is possible that SQLite might one day be recoded in Rust. Recoding SQLite in Go is unlikely since Go hates assert(). But Rust is a possibility. Some preconditions that must occur before SQLite is recoded in Rust include:
- Rust needs to mature a little more, stop changing so fast, and move further toward being old and boring.
- Rust needs to demonstrate that it can be used to create general-purpose libraries that are callable from all other programming languages.
- Rust needs to demonstrate that it can produce object code that works on obscure embedded devices, including devices that lack an operating system.
- Rust needs to pick up the necessary tooling that enables one to do 100% branch coverage testing of the compiled binaries.
- Rust needs a mechanism to recover gracefully from OOM errors.
- Rust needs to demonstrate that it can do the kinds of work that C does in SQLite without a significant speed penalty.
1. Rust has had ten years since 1.0. It changes in backward compatible ways. For some people, they want no changes at all, so it’s important to nail down which sense is meant.
2. This has been demonstrated.
3. This one hinges on your definition of “obscure,” but the “without an operating system” bit is unambiguously demonstrated.
4. I am not an expert here, but given that you’re testing binaries, I’m not sure what is Rust specific. I know the Ferrocene folks have done some of this work, but I don’t know the current state of things.
5. Rust as a language does no allocation. This OOM behavior is the standard library, of which you’re not using in these embedded cases anyway. There, you’re free to do whatever you’d like, as it’s all just library code.
6. This also hinges on a lot of definitions, so it could be argued either way.
Why can't `if condition { panic(err) }' be used in go as an assert equivalent?
I wonder if the hype helps rust being a better language
At this point I wish the creators of the language could talk about what rust is bad at.
“None of the safe programming languages existed for the first 10 years of SQLite's existence. SQLite could be recoded in Go or Rust, but doing so would probably introduce far more bugs than would be fixed, and it may also result in slower code.”
Modern languages might do more than C to prevent programmers from writing buggy code, but if you already have bug-free code due to massive time, attention, and testing, and the rate of change is low (or zero), it doesn’t really matter what the language is. SQLIte could be assembly language for all it would matter.
I think it’s more interesting that DuckDB is written in C++ and not rust than SQLite.
SQLite is old, huge and known for its gigantic test coverage. There’s just so much to rewrite.
DuckDB is from 2019, so new enough to jump on the “rust is safe and fast”
if they write it on modern C++ then its alright tbh
SQLite works great in Fil-C with minimal changes.
So, the argument for keeping SQLite written in C is that it gives the user the choice to either:
- Build SQLite with Yolo-C, in which case you get excellent performance and lots of tooling. And it's boring in the way that SQLite devs like. But it's not "safe" in the sense of memory safe languages.
- Build SQLite with Fil-C, in which case you get worse (but still quite good) performance and memory safety that exceeds what you'd get with a Rust/Go/Java/whatever rewrite.
Recompiling with Fil-C is safer than a rewrite into other memory safe languages because Fil-C is safe through all dependencies, including the syscall layer. Like, making a syscall in Rust means writing some unsafe code where you could screw up buffer sizes or whatnot, while making a syscall in Fil-C means going through the Fil-C runtime.
The point about bounds checking in `safe' languages is well taken, it does prevent 100% test coverage. As we all agree, SQLite has been exhaustively tested, and arguments for bounds checking in it are therefore weakened. Still, that's not an argument for replicating this practice elsewhere, not unless you are Dr Hipp and willing to work very hard at testing. C.A.R. Hoare's comment on eliminating runtime checks in release builds is well-taken here: “What would we think of a sailing enthusiast who wears his life-jacket when training on dry land but takes it off as soon as he goes to sea?”
I am not Dr Hipp, and therefore I like run-time checks.
Two previous, and substantial, discussions on this page:
https://news.ycombinator.com/item?id=28278859 - August 2021
https://news.ycombinator.com/item?id=16585120 - March 2018
One thing I found especially interesting is the section at the end about why Rust isn’t used. It leaves open the door and at least is constructive feedback to the Rust community
The fact that a C library can easily be wrapped by just about any language is really useful. We're considering writing a library for generating a UUID (that contains a key and value) for reasons that make sense to us and I proposed writing this in C so we could simply wrap it as a library for all of the languages we use internally rather than having to re-implement it several times. Not sure if we'll actually build this library but if we do it will be in C (I did managed to get the "wrap it for each language" proposal pre-approved).
You can expose a C interface from many languages (C++, Rust, C# to name a few that I've personally used). Instead of introducing a new language entirely, it's probably better to write the library in one of the languages you already use.
It is. You can also write it in C++ or Rust and expose a C API+ABI, and then you're distributing a binary library that the OS sees as very similar to a C library.
Occasionally when working in Lua I'd write something low-level in C++, wrap it in C, and then call the C wrapper from Lua. It's extra boilerplate but damn is it nice to have a REPL for your C++ code.
Edit: Because someone else will say it - Rust binary artifacts _are_ kinda big by default. You can compile libstd from scratch on nightly (it's a couple flags) or you can amortize the cost by packing more functions into the same binary, but it is gonna have more fixed overhead than C or C++.
These points strike me:
Turso:
https://algora.io/challenges/turso "Turso is rewriting SQLite in Rust ; Find a bug to win $1,000"
------
- Dec 10, 2024 : "Introducing Limbo: A complete rewrite of SQLite in Rust"
https://turso.tech/blog/introducing-limbo-a-complete-rewrite...
- Jan 21, 2025 - "We will rewrite SQLite. And we are going all-in"
https://turso.tech/blog/we-will-rewrite-sqlite-and-we-are-go...
- Project: https://github.com/tursodatabase/turso
Status: "Turso Database is currently under heavy development and is not ready for production use."
If the branch is never taken, and the optimizer can prove it, it will remove the check. Sometimes if it can’t actually prove it there’s ways to help it understand, or, in the almost extreme case, you do what I commented below.
Yeah I don't understand the argument. If you can't convince the compiler that that branch will never be taken, then I strongly suspect that it may be taken.
> Safe languages insert additional machine branches to do things like verify that array accesses are in-bounds. In correct code, those branches are never taken. That means that the machine code cannot be 100% branch tested, which is an important component of SQLite's quality strategy.
This is annoying in Rust. To me array accesses aren't the most annoying, it's match{} branches that will never been invoked.
There is unreachable!() for such situations, and you would hope that:
is recognised by the Rust tooling and just ignored. That's effectively the same as SQLite is doing now by not doing the check. But it isn't ignored by the tooling: unreachable!() is reported as a missed line. Then there is the test code coverage including the standard output by default, and you have to use regex's on path names to remove it.A more direct translation of the sqlite strategy here is to use get_unchecked instead of [], and then you get the same behaviors.
Your example does what [] does already, it’s just a more verbose way of writing the same thing. It’s not the same behavior as sqlite.
But if you don't have the bounds checks in machine code, then you don't have bounds checks.
I suppose SQLite might use a C linter tool that can prove the bounds checks happen at a higher layer, and then elide redundant ones in lower layers, but... C compilers won't do that by default, they'll just write memory-unsafe machine code. Right?
because Rust isnt out yet back then????
> Rust needs to mature a little more, stop changing so fast, and move further toward being old and boring.
Talking about C99, or C++11, and then “oh you need the nightly build of rust” were juxtaposed in such a way that I never felt comfortable banging out “yum install rust” and giving it a go.
Other than some operating systems projects, I haven’t run into a “requires nightly” in the wild for years. Most users use the stable releases.
(There are some decent reasons to use the nightly toolchain in development even if you don’t rely on any unfinished features in your codebase, but that means they build on stable anyway just fine if you prefer.)
I’d be curious to know what the creators of SQLite would have to say about Zig.
Zig gives the programmer more control than Rust. I think this is one of the reasons why TigerBeetle is written in Zig.
> Zig gives the programmer more control than Rust
More control over what exactly? Allocations? There is nothing Zig can do that Rust can’t.
I think zig generally composes better than rust. With rust you pretty much have to start over if you want reusable / composable code, that is not use the default std. Rust has small crates for every little thing because it doesn't compose well, as well to improve compile times. libc in the default std also is major L.
> Nearly all systems have the ability to call libraries written in C. This is not true of other implementation languages.
From section "1.2 Compatibility". How easy is it to embed a library written in Zig in, say, a small embedded system where you may not be using Zig for the rest of the work?
Also, since you're the submitter, why did you change the title? It's just "Why is SQLite Coded in C", you added the "and not Rust" part.
The article allocates the last section to explaining why Rust is not a good fit (yet) so I wanted the title to cover that part of the conversation since I believe it is meaningful. It illustrates the tradeoffs in software engineering.
> Otherwise please use the original title, unless it is misleading or linkbait; don't editorialize.
From the site guidelines: https://news.ycombinator.com/newsguidelines.html