Logo

In 2029, will any AI be able to construct "reasonably" bug-free code of >= 10k LOC from a natural language specification? (Gary Marcus benchmark #4)

$26,924 volume on Manifold

Prices here are free to read and need no account. A free account lets you bookmark this market, follow the traders moving it, and opt in to the weekly digest. We never hold funds or place trades.Create a free account →

manifold
In 2029, will any AI be able to construct "reasonably" bug-free code of >= 10k LOC from a natural language specification? (Gary Marcus benchmark #4)
92.2%$0.92
Loading chart...

How this market resolves

The fourth question from this post: https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things The full text is: "In 2029, AI will not be able to reliably construct bug-free code of more than 10,000 lines from natural language specification or by interactions with a non-expert user. [Gluing together code from existing libraries doesn’t count.]" Judgment will be by me, not Gary Marcus. Ambiguous whether this means start or end of 2029, so I have set it for the end. For this question I am not using the exact text of the question, because I think "bug-free" is 1. silly 2. untestable. I will instead accept if it produces code of >=10k LOC with <= the number of bugs in an implementation by a human (many small bugs for some significant bugs will unfortunately be down to my subjective impression of whether it's "better") I am also ignoring the "no gluing libraries together" requirement, because I don't know what he means. Does he want an AI that writes 10k LOC of assembly? I will accept code that is calling/using libraries at <= the rate that normal human programmers do. Sep 16, 2:26pm: Some additional clarifications: If there was a benchmark that, for instance…

Source: Manifold rules — excerpt of the text the venue publishes.

Selected BetIn 2029, will any AI be able to construct "reasonably" bug-free code of >= 10k LOC from a natural language specification? (Gary Marcus benchmark #4)
M
Trading via Manifold
Market Volume$26.9K
Yes
92.2%
No
7.8%
Trade
No whale trades found for this event.