Hacker news

  • Top
  • New
  • Past
  • Ask
  • Show
  • Jobs

Latent Lattice Language Models (https://shivamduggal4.github.io)

56 points by E-Reverance 3 days ago | 5 comments | View on ycombinator

augment_me 2 days ago |

Seems like just an extension of Short-conv in the linear attention methods where we instead carry the EMA of the entire sequence instead of nearest 4 tokens.

It would make sense in my opinion to do the evals based on the reason for needing Short-conv in the architectures that use it in the first place, where it was added to support exact retrieval/needle-in-haystack when we compress the attention state.

It's odd to make this arch extension and not test beyond a 2k context window, or with the needle-in-haystack benchmark.

In my head this should fail because the EMA cannot carry this granular information

royal__ 2 days ago |

How significant is a .98 decrease in perplexity. I'm a researcher in this field and I'm having trouble interpreting the implication of this.

tkslucas 2 days ago |

Interesting approach to scale compute, never thought of using the per layer state like that

amitport 2 days ago |

Isn't that "Infini-attention"? https://arxiv.org/abs/2404.07143 (2024)

bionhoward 2 days ago |

I am so excited for RNNs to be cool again