OpenAI Researchers Find That Even the Best AI Is "Unable To Solve the Majority" of Coding Problems

The trick is that you don't want to generate a lot of code all at once. I tend to use it for making specific functions, or doing boring tasks like creating SQL schemas based on JSON input. I can validate these tasks pretty easily, but it saves me a lot of time looking stuff up or writing boilerplate.

I also find that different languages also work better in this context. For example, I primarily work with Clojure and it's a functional language where data is immutable. The contract for most functions is a plain data structure without any state or mutable references to the outside. This makes it easy to test functions in isolation to ensure they're doing what's intended. For example, I had DeepSeek write a function to extract image urls from Markdown for me just the other day.

    
(defn extract-image-links
  "Extracts all image links including reference-style images"
  [markdown]
  (let [;; Extract reference definitions
        ref-regex #"(?m)^\[([^\]]+)\]:\s+(https?://\S+)$"
        refs (into {} (map (fn [[_ ref url]] [ref url])
                           (re-seq ref-regex markdown)))
        ;; Extract all image tags
        image-regex #"!\[([^\]]*)\]\(([^\)]+)\)|!\[([^\]]*)\]\[([^\]]*)\]"
        matches (re-seq image-regex markdown)
        ;; Process matches
        urls (keep (fn [[_ _ inline-url _ ref]]
                     (cond
                       inline-url (string/trim inline-url)
                       ref (get refs (string/trim ref))))
                   matches)]

    (set urls)))

  

It's easy to follow code that's fairly idiomatic, and I can easily test this function by passing some Markdown through it and seeing whether it's giving me expected results. The most annoying part about writing it by hand would've been crafting the regex which DeepSeek managed to do correctly.

What aspect of it do you think users are deficient in that they’re missing 95% of its potential? Prompt engineering?