87 points by yonasb 3 days ago | 52 comments | View on ycombinator
colek42 3 days ago |
devkumardev2001 about 22 hours ago |
On Bnjoroge’s question, compare the total test count in the merged report with npx playwright test --list on the same commit. That gives you a simple check that the expected tests are represented in the results.
Two lessons from running Playwright in Kubernetes: CPU throttling can cause timeouts that look like flaky tests, so monitor throttling before increasing retries. And keep flaky tests in a separate, non-blocking job rather than letting an optimizer remove them. Otherwise, you improve the metric without fixing the underlying issue.
Disclosure: I work on TestDino, a Playwright reporting tool.
emscb 2 days ago |
bekacru 2 days ago |
ansh25lehri 1 day ago |
Also, since some wins come from app code (like the better-auth fix), how do you decide when a change outside the workflow/test files is in scope? Beyond the test-parity check, is there a guardrail on how much business code the agent can touch?
Shexfer 1 day ago |
Is the agent diffing the test bodies as well as the run results? A guard on test-file changes would make the speedup claim much easier to trust.
jstrasner 3 days ago |
codedrift_ 1 day ago |
How does StarSling handle that drift? Does the loop keep watching a workflow after a PR merges, and will it catch a regression on a win it already shipped?
esafak 3 days ago |
poxlr 2 days ago |
sochs 3 days ago |
brunoarueira 2 days ago |
jbqueries 2 days ago |
Actual problem tho, congrats on the launch!
MisterMunchkin 1 day ago |
dotmike 3 days ago |
d-sanoj 3 days ago |
tristenharr 2 days ago |
buildwrangler 3 days ago |
gkoberger 3 days ago |
What are some of the most interesting things the AI has caught + fixed so far?
kinfish 3 days ago |
Three things matter.
1 -- How many non-conflicting changes can actually be made at one time (Fan Out)
2 -- How long a critical path takes
3 -- How many times you have to run that critical path. Most orgs run it 2-3x, once before the PR in the agent loop, once in CI, and another in the MQ
This is why most orgs cant get past about 12 PRs an hour.