Python @programming.dev Jason Novinger @programming.dev 3mo ago

Python Performance: Why 'if not list' is 2x Faster Than Using len()

blog.codingconfessions.com Python Performance: Why 'if not list' is 2x Faster Than Using len()

Discover why 'if not mylist' is twice as fast as 'len(mylist) == 0' by examining CPython's VM instructions and object memory access patterns.

Technology @lemmy.world abhi9u @lemmy.world 3mo ago

blog.codingconfessions.com /p/python-performance-why-if-not-list

21 comments

One of the principles of the Pythonic style is: simple is better than complex.

But is it when you consider ambiguity and expressiveness too?

len(mylist) tells me it's definitely a list and not null or whatever. It also tells me the intent of whoever writes the code was checking length.

If it requires a long article to explain, I'd certainly be hesitant to use and prefer. The question is, is it surprising or intuitive that the truthy becomes a length check. What do you want to express with your code. And who will read it, and what expectations and requirements do you want to require of them.

For sequence type objects, such as lists, their truth value is False if they are empty.
- len(mylist) tells me it’s definitely a list and not null or whatever
  
  Do you know what else can tell you it's a list? Type hinting.
  
  If I care about the distinction between None and an empty list, I'll make separate checks. len(None) raises an exception, and most of the time that's not what I want when checking whether there's something in the list, so not x is generally preferred, especially when type hinting is used consistently enough to be reasonably sure I'm actually getting a list or None. If I get something else, that means things got really broken and they'll likely get an exception alter (i.e. when doing anything list-like).
  
  For sequence type objects, such as lists, their truth value is False if they are empty.
  
  That's generally exactly what I want. len(x) == 0 doesn't tell you it's a list, it just tells you it has __len__. So that could be a dict, list, or a number of other types that have a length, but aren't lists.
  
  Don't rely on checks like that to tell you what type a thing is.
- It's just how pythonic code is written. All python developers know that not and bool can indicate that variable is either empty or null. They will also use the in operator and other python specific syntax.
- foo = "potatoes!"
  
  len(foo)
  
  will give you the number of characters in the string. It's not a surefire way to find a list at all!
  
  Same with dictionaries, iterators (will consume the iterator!), and anything else that has a length.
  
  I've actually had the string instead of list issue a number of times, because sometimes things get off and it's hard to track down where things went wrong.
  
  For this reason, I think type hints are the way to go. Use them consistently w/ a static analysis tool like pyright and you'll catch a lot of these issues before running into cryptic error messages.
I don't know about others ... but I'm not using Python for execution speed.

Typically the biggest problem in a program is not 100 million calls of len(x) == 0. If it was, the interpreter could just translate that expression during parsing.
- I mean, nobody uses Python for execution speed precisely because it is so slow.
- Even so, not x is a pretty nice-to-read pattern, and it's nice that it's faster than the less nice len(x) == 0. I generally do not care to distinguish whether a list is None or empty, I just want to know if there's something there. If I care, then I'll typically separate those checks anyway: if x is None: ..., if len(x) == 0: ....
  
  I prefer the clarity of len(x) == 0 personally; it's a pretty low syntactic penalty to clarify intent.
  
  Obviously, if your variable names are good and you're consistent about your usage not list can be fine too, but it is a very JavaScript-y thing to me to be that vague and/or rely on "truthiness."
- The interpreter can't make the replacement until it's about to execute the line as __bool__ and __len__ are both (Python's equivalent of) virtual functions, so it's got to know the runtime type to know if the substitution is valid. Once it's that late, it'll often be faster to execute the requested function than work out if it could execute something faster instead. Even with type hints, it's possible that a subclass with overridden methods could be passed in, so it's not safe to do anything until the real runtime type is known.
  
  Once there's a JIT involved, there's an opportunity to detect the common types passed to a function and call specialised implementations, but I don't think Python's JIT is clever enough for this. LuaJIT definitely does this kind of optimisation, though.
  
  Hm... I'll admit I wasn't awkward of the .__len__ function. ~~However, does this mean it's possible to write a len(x) == 0 that's diverges from not x?~~
  
  ~~If not, then the substitution is still valid (and not presumably also considers the same fundamental. If so, that's kind of silly.~~
  
  EDIT: I missed the part of your comment about .__bool__ ... so yeah in theory you could have something where these two operations are not equivalent.
  
  Arguably, you could just say that's pathological and invalid. Then, still have an optimized path to prefer not .__bool__() if .__len__() == 0 is the comparison you'd be making. Even with the extra interpreter check during evaluation, that would quite possibly be faster if the overhead is truly high.
  
  EDIT 2: you'd probably need a little bit more overhead than a straight substitution anyways because you need to maintain the semantic of "if this thing is None it's not valid if the syntax was originally len(x)."
- This. I rarely boot up Python for the tasks I need to do, and if they are, they are one of the following:
  
  Assistant code for other coding language
  
  Throwaway script
  
  Prototype before using a faster language
  
  Assuming an equivalent package is produced, what's the maintenance cost (factoring in coder availability) difference between the Python vs faster language implementations?
  
  ^^ therein lies the rub
  
  Reminds of the expression, premature optimization is the root of all evil
  
  if not swimming in funding, might be a darwinic move to choose the faster language and have to code everything yourself from scratch
Makes perfect sense. If you're checking if a collection is empty you don't need to know its exact size. Getting the size can be very inefficient in collections like linked lists or trees, if you have to follow all nodes. To check if it's empty, all you need fo know if at least one item exists. If one does, there's no point counting the rest.

People who don't understand the difference will probably not understand the difference between passing a list and passing an literator/generator to any() .
While the 2nd approach is not wrong, the first method is considered more Pythonic. Many people don’t agree, but I’ve already put forward my points in a previous article on that debate.

Does Pythonic mean best practice in the python community or "good python"?

If "many people don't agree", how can they claim it to be "pythonic"? Isn't that contradictory?
- "Many" isn't the same as "most," though I don't think there's any way to know what "most" is.
  
  But here's my reason for agreeing w/ the OP:
  
  not x checks both None and emptiness, which is usually desired
  
  len(x) == 0 will raise an exception if x is null
  
  with type hinting, those should be the only two reasonable options
  
  It's nice that it's slightly faster, though performance has absolutely nothing to do w/ my preference, though it does have something to do with my preference for avoiding exceptions for non-exceptional cases.
lol this whole article and this thread is a bunch of bikeshedding I haven't had the privilege of enduring in some weeks.
One more ridiculous fact about the Python to know Thank you for sharing.

21 comments