279 points by ghostfoxgod 1 day ago | 72 comments | View on ycombinator
anacrolix about 21 hours ago |
sgarland about 18 hours ago |
This is true of every RDBMS I am aware of. It would be horribly inefficient to constantly run page merges in order to be able to free some of them.
Whenever I read things on posts that are a. discussed in documentation b. should be apparent from first principles, I immediately lose faith in the product.
sneilan1 about 16 hours ago |
> While raw files start to beat SQLite above ~100 KiB, we compress blobs before writing them. And thankfully, 99% of our compressed values sit well below 100 KiB.
My question is, were the other databases not tried with compression on the application side?
lukax about 19 hours ago |
bob1029 about 22 hours ago |
An alternative to faster blob storage is to represent the blob's schema in SQL directly.
If most blob updates are incremental you can eliminate a significant amount of storage IO and garbage collection this way.
One SQLite database per blob type with a metadata SQLite db that tracks each. We could leverage powerful AI reasoning models to produce SQLite schemas and virtual table adapters for each encountered JSON schema type.
gkamal about 19 hours ago |
munch-o-man about 10 hours ago |
https://github.com/afreidah/s3-orchestrator https://s3-orchestrator.munchbox.cc
self hosted service that acts as an orchestration layer between x-number of s3/blob storage backends, handles replication, failover, encryption, compression, integrity checking, and backend pressure analysis to optimize reads/writes to the most efficient available backend that meets the requirements of the request.
Threw a star on yours so I can remember to look at it deeper later
CyrusGrissom about 22 hours ago |
matja about 21 hours ago |
articulatepang about 18 hours ago |
These are user preferences for UI setup, and they have a lot of write traffic. But probably each user changes their preferences only by a little bit each time. Their 1MB writes would go down to just the small diffs.
They could store the diffs and transparently apply them before sending to the client. After some timeout or maybe after a certain number of diffs have accumulated, squash them all and write back.
They’d still need a server and custom code to handle this but it would probably reduce the database load and SSD writes by a lot. And let them just use something off-the-shelf for storage.
vmikhailov about 17 hours ago |
grep_it about 12 hours ago |
feverzsj about 23 hours ago |
Blobtits about 20 hours ago |
martinjc about 21 hours ago |
Kalanos about 21 hours ago |
zodwick about 22 hours ago |
adsharma about 9 hours ago |
Kalanos about 21 hours ago |
murkt about 21 hours ago |
Bad fault tolerance, no words on replication or backup. Offline rebalance... meh. I am not sure how I feel about their choice of Redis protocol.
robinduckett about 21 hours ago |
ltbarcly3 about 11 hours ago |
Filesystem! I knew there was some obscure way to do this.
ratherlongname about 20 hours ago |
nodesocket about 17 hours ago |
redwood about 18 hours ago |
_s_a_m_ about 21 hours ago |
EGreg about 17 hours ago |
https://github.com/Safebots/Safecloud
It can also stream videos to any domain, like a decentralized YouTube. Here is a demo:
https://youtu.be/glWQM0BIcDQ?si=VV7YFiElGKBC82Bv
This is what we encourage to store in it:
https://safebots.ai/datasets.html
Oh, and it is also compatible with S3 bucket protocol, IPFS, and even MySQL protocol so it can replace your S3 bucket or database with a distributed database — just like that!
PS: if you’re curious, it uses a columnar store and SQLite’s query planner, to be as efficient as possible.
arbor-group about 18 hours ago |
patryk_builds about 20 hours ago |
foolgry about 22 hours ago |
aman179102 about 21 hours ago |
wangguode about 19 hours ago |
agline_in about 18 hours ago |
vrox about 23 hours ago |
dklabsdev about 16 hours ago |
zorvelaieffects about 14 hours ago |
routebraid about 16 hours ago |
mesm7903 about 23 hours ago |
MalgorzataKowal about 22 hours ago |
I wrote https://github.com/anacrolix/possum for this exact use case. It has a few rough edges but it absolutely hauls ass for caching on disk. Incidentally it uses sqlite for transaction/TTL metadata which is more suited to sqlite but it's still the bottleneck.
sqlite transaction commit overhead is too high and uncompetitive without batching but it's a design flaw specific to being multiprocess.