[Troubleshooting] Random EOF shutdowns from Klipper.
[Troubleshooting] Random EOF shutdowns from Klipper.
Hi everyone, a week ago my printer (heavily modified Neptune 3) started randomly shutting down in the middle of prints. I come back to a print with the "Klipper reports: SHUTDOWN / Lost communication with MCU 'mcu'" error message.
The printer has been "under construction" for the last couple of weeks, but it has been in varying states of "working" for most of the time - working well enough for me to print the parts I needed to get it back to "fully operational". During this time, the printer never shut down like it is now.
Only once I started making little cosmetic changes did the problem present itself. I was running a known-good print, and I got the above error twice (first time after ~2 hours, second time after ~1 hour) before I got a successful print off of it. This was last week.
After this successful print, I continued other prints with no issues. After a day or two with no problems, an hour long print threw the error at me four consecutive times between 10-45 minutes into the print. This is when I started looking into my klippy log and found some relevant articles citing things like EMF interference, bad power supplies, faulty cables etc. I realized that one of the changes I had made rerouted the printer USB cable right around the Z-stepper, so I rerouted it to how it was originally and immediately managed a successful print. This was 5 days ago.
After moving that cable I had no issues with printing several-hour long prints... until last night. I had been printing all day, then the problem came back. After one print finished, I queued up another print with a plate full of parts, it failed after 1.5 hours. Tried the same print again, failed in 30 minutes. I re-sliced to only a handful of parts to see if I could get those to print before the error occurs, and it's failing 15 minutes into the print.
The printer power supply is the unit that came with the Neptune, and it isn't powering anything besides stock hardware (exception being the SKR mini board), so I don't think it's that. The pi is on a quality unit. The USB cable has been working for a long time so I also don't suspect that, but I'm probably going to buy a new one today just to be sure. I adjusted my enclosure setup so that the Pi and SKR are able to get cool air (at one point had a personal fan pointing at the open electronics box, still failed).
Here is a link to my most recent klippy log (abridged to the start of the last failed print). I'm not very familiar with reading through this and finding oddities, but I do think it's strange that it seemed to load my preheat script in the middle of printing right before the EOF error. (It should be noted that this preheat script was made 1 or 2 failed prints before this most recent one, so it isn't the source of the error as prints were failing before the script was made). If there's anything I'm missing or something else I can try, please let me know!
Edit: While typing this post, I was running the same failed print without filament and both heaters turned off. It ran for about 45 minutes (most recent failure occurred at 12 minutes) so I cancelled the print and started it again with heaters turned on, still without filament. It again ran for about 45 minutes, so I again cancelled it and started the print again, this time with filament loaded. It failed in 5 minutes.
Edit 2: A test print with heaters on and no filament failed after 1h8m. So it isn't an issue with extruding filament.
Edit 3: New cable with the 5v leads taped off per @SzethFriendOfNimi@lemmy.world's advice. Ran the print without filament until completion. Reloaded the same file with filament, print ran without issue until the 1h14m mark, at which point I tapped my Klipperscreen device to wake up the screen, and as soon as it displayed the status, the printer errored out. This can't be a coincidence, can it? Whenever the print goes unmonitored for a long time, it fails as soon as I do something (load mainsail, turn on the klipperscreen) to check the status of it.
Another thing to consider is making a data only USB ( I use a sliver of electrical tape over the 5v+ line ) to stop any power draw between the usb port and the printer.
And, of course, check that you have a good power brick since shoddy ones always cause issues with pis.
To be honest sounds like you’ve got a gremlin jumping from one piece of equipment to another so maybe prayer, voodoo, holy water and or appeals to the deity of universal fundamental force of your choosing could help too.
I'd never heard of the blocking the 5v trick before. If the current test fails I might give that a go.
Power brick for the pi is good. It isn't the official one, but a highly reviewed one, and I never get voltage warnings like I did on the old one.
To be honest sounds like you’ve got a gremlin jumping from one piece of equipment to another
You must have seen my other posts.
How old is your Pi? I had the same issues on a generic board (Le Potato) but upgrading to a Pi 4 Model B fixed it for me. I have both my Neptune 2S (modded) and Neptune 3 Pro (stock) running on the same Pi with no issues.
Definitely change your USB cable. If you've got some complex files, maybe the cable can't handle the bandwidth (experienced this back when I was first starting out)
The pi is an 8gb Pi 4, maybe 3 months old. Usage never goes above ~15% for both cpu and mem during prints. Should be rock solid. This replaced my Pi 3 which was dropping USB connection with 3 webcams and klipperscreen running. (Currently only running one webcam.)
I'm heading to the store now for a new USB cable, but I'm wondering if it might be something to do with my extruder motor? Since starting the OP, I've run the same test print 4 times, first 2 ran for ~45 minutes with no filament, 3rd one with filament errored in 5 minutes, I'm currently running test 4 and it's gone 25 minutes without filament.
The only thing I can think of with the extruder is thermal runaway, but Klipper would tell you if that was the case. Have you tried with no camera plugged in? If new USB cables aren't working, it might be worth starting from scratch and rebuilding your Klipper instances. If a clean Klipper install (with no webcams or anything) is breaking, then it's likely going to be a hardware issue.
How are you powering the pi itself? Wondering if the wakeup draw from your screen is enough to make it unstable, 4 I believe having higher power drawn than the 3 if you're using the same power supply. Pi isn't oced either?
Quick Edit: I run my v2.4 on a lepotato with oodles of usb attachments (camera, multiple mcus, wifi and a screen. Replacing the mcu connections with a usb-canbus bridge when the heatwave ends) but with it connected directly to a meanwell 5v5a PSU into the gpio header, have never had communication issues to the octopus pro I use. Skr mini on the other printer is connected to a laptop host so it definitely has enough power, did have some odd issues with the skr mini and having an accelerometer connected to the spi header on boot and separately the 24v supply becoming loose that I fixed by crimping ferules onto the supply wires when I added a molex connector to make taking the printer out of its enclosure easier.
Pi is not OC'ed, but I did just realize that it's running on a power supply that came in a cheap starter kit and not the nicer one I'd ordered, so maybe that's an issue. You would think the Pi would report under voltage if that were the case though, no? When Klipper errors out, the pi doesn't shut off or report any issues.
I will say that I've completed two 4 hour long prints with all external USB devices disconnected, and when I tried a duplicate of one of the completed prints with the klipperscreen device (on a new nice cable) and my webcam (connected but not enabled via crowsnest) it failed after 3 hours.
You'd think right, anecdotally when I was meaaign with undervolting on my desktop, you'd get instability or crashes at the limits without useful errors. Could think maybe the processor isn't getting enough power and that's causing instability? Found a klipper thread that about losing connection with the mcu which totally can show up as an EOF error.