169 points by eatonphil 8 days ago | 57 comments | View on ycombinator
reticulates 3 days ago |
4ndrewl 3 days ago |
My understanding is that modern databases will take advantage of that if you create an index of those fields - ie it'll read from the index, not the records.
j0shk0 3 days ago |
netcraft 3 days ago |
remywang 3 days ago |
PAX’s “row group” has the same size as a disk page, whereas Parquet row groups are much larger so a page stores only data from the same column. I’ve been trying to track down the origin of this change in design but haven’t been able to find the right paper
Leftium 2 days ago |
- ~200X decrease in memory usage
- Moby Dick went from 700MB to about 1MB
---
Another practical application of optimization with columns are used in game dev with entity component systems (ECS). The cache locality can really help performance: https://en.wikipedia.org/wiki/Entity_component_system
Peteragain 3 days ago |
makmanalp 3 days ago |
“Column-Stores vs. Row-Stores: How Different Are They Really?”, 2008
https://15721.courses.cs.cmu.edu/spring2016/papers/p967-abad...
kinleyd 3 days ago |
postexitus 3 days ago |
krawcu 3 days ago |
> [...]
> [24] People have tried to build one system that does both in the past (HTAP, or hybrid transactional/analytical processing), but those attempts generally didn't work out. Some newer systems seem to be bridging the gap, however. CedarDB, the commercial version of Umbra (a research system from TU Munich), runs transactions and analytics on just one engine and one copy of the data. Databricks' Lakebase keeps Postgres (a row store) as the engine for transactional queries, but its storage layer rewrites the same data into columnar formats in the background, allowing an analytical engine to query it.
The footnote says "those attempts generally didn't work out" but then mentions Lakebase and CedarDB as newer efforts. Can anyone speak to whether or not CedarDB, Lakebase (and any others?) are achieving the fabled transactions + analytics in one database? Should we expect, at some point in the near future, for transactions + analytics to be standard in all databases, or is combining them a niche / esoteric thing with many tradeoffs?