As a beginner, how should I go about learning difficult concepts?
I'm trying to learn programming and something I struggle with the most is trying to separate code mentally into chunks where I can think through the problem. I'm not really sure how to describe it other than when I read a function to determine what it does then go to the next part of the code I've already forgotten how the function transforms the data and I get stuck trying to figure out the solution. So instead I'll often cludge something together just to make it work but I don't feel like I made any progress. Has anybody else run into this issue where they struggle with abstracting code from text to mental instructions?
Edit: Thank you all for the suggestions and advise. I wish I could reply to everyone but there's been a lot of good information given and I have some ways now to try and train my brain to think about how to break down the code. It's also a little reassuring knowing I'm not the first to have these same struggles.
I'm not really sure how to describe it other than when I read a function to determine what it does then go to the next part of the code I've already forgotten how the function transforms the data
This sounds to me like you could benefit from mentally using the information hiding principle for your functions. In other words: Outside of the function, the only thing that matters is "what goes in?" and "what comes out?". The implementation details should not be important once you're working on code outside of that function.
To achieve this, maybe you could write a short comment right at the start of every function. One to two sentences detailing only the inputs/output of that function. e.g. "Accepts an image and a color and returns a mask that shows where that color is present." if you later forget what the function does, all you need to do is read that one sentence to remember. If it's too convoluted to write in one or two sentences, your function is likely trying to achieve too much at once and could (arguably "should") be split up.
Also on a different note: Don't sell your ability to "cludge something together" short. If you ever plan to do this professionally or educationally, you will sadly inevitably run into situations where you have no choice but to deliver a quick and dirty solution over a clean and well thought out one.
I have the same problem as OP, and this is how exactly how I cope! If the details are irrelevant, than you don’t need to keep them in short term memory
It's a lot to take in at first, but the problem isn't that you're really doing something wrong or that you're lacking some piece of information. You just need to actually write more code. It will help you understand why something is set up the way that it is, and also help things make a lot more sense. Once you have that foundation of experience, a lot of other things fall into place on their own for lack of a better explanation. In other words: practice makes perfect
How long would you say it took you before getting a fundamental understanding? I ask because I've been at this on and off for years but I always end up quitting because after a few months I just don't see any progress happening. I'm still forgetting things I learned 3 or even 4 times like how to do a for each loop.
But as you said it just takes practice. I've tried to find challenges that I could do and everything is just so overwhelming I have idea where to start. I see tutorials say to make a tic tac toe game or a calculator or to contribute to open source code. Which is good I suppose but all of it feels too advanced and I get lost on how to begin. For reference I'm currently learning html, css, and JavaScript. Advent of code was okay when I tried but once I got past the first handful of challenges it quickly went way over my head with sorting algorithms and how to make maps out lists for the elves to move or whatever.
I’m still forgetting things I learned 3 or even 4 times like how to do a for each loop.
I have been programming for decades now and still have to look up how an if statement works in bash - or other similar things, especially when switching between languages. It takes 5 seconds to look up and remember so I would not bother worrying about it. Far better to know when you need to use an if or for loop and quickly look up the syntax then to know the syntax but not when to use it.
I see tutorials say to make a tic tac toe game or a calculator or to contribute to open source code. Which is good I suppose but all of it feels too advanced and I get lost on how to begin.
Break problems down into simpler problems, then break those down into simpler problems until you have a trivial problem you can solve. Then build up from there. Like take a tic tac toe game - lots of things to consider that can all be dealt with in isolation. Like rendering the game to the screen, that is one problem you can start with, and can be broken down even further to maybe how to draw a grid to the screen, which again can be broken down to how to draw a box or line.
You might even want to look at the book " Think Like a Programmer: An Introduction to Creative Problem Solving by V. Anton Spraul" which goes into this way of thinking in more detail.
Contributing to existing projects as an introduction is very hit or miss. Project and task complexity vary immensely, supporting docs and guidance are most often non existent.
I love Advent of code as a concept, but I agree - at least the ones I worked on - have a steep difficulty/scope curve.
Something like a tic tac toe game is great because it is visual, interactive, and iterative as well as relatively small in scope. Any project you have a personal interest in and that has some or most of those properties is great.
The Web technologies HTML css and js are great to get into programming but I feel like it's bad for teaching software development as software engineering, because it doesn't guide towards or ensure structuring and good practices. Which I thought you were talking about at first, but I guess you're not at that point yet with on and off beginnings.
Going back to your original description, functions encapsulate a work unit. Use it to name and define behavior so that you can combine and hide away complexity in a defined and obvious manner. Separation makes overall complexity manageable because you can look at subsection of it.
How did your tic tac toe game go? Was that something that worked out well?
Have you looked at other projects? For example other tic tac toe implementations and how they did it? Which can be hit or miss in quality and readability of course.
In my opinion the best way to learn, or environment to learn, is teaching or/and guidance through a good senior. The second best is interest and personal projects. Even if it's small hacks or projects, and even "unfinished" projects can give experience and knowledge.
How long would you say it took you before getting a fundamental understanding?
I would say years, as with any complex activity.
I'm still forgetting things I learned 3 or even 4 times like how to do a for each loop.
You can forget in 2 different ways:
Forget how to use something, so you need to look how to do it.
Forget that something exists, so you cannot even look for it because you are not aware it's a possibility.
You will forget-1 everything which you don't use on a daily basis. That's what internet is for. Forgetting in the 2-nd sense is much more rare and you should do something if that's the case.
all of it feels too advanced and I get lost on how to begin
This is a bias most of us have, you overlook how easy is for you to do things that previously were impossible and focus on how hard are the things you still don't know how to do. And computing is so complex right now that there always be "infinite" things you don't know.
Try showing what you know to someone who doesn't know how to code and you will get an idea of how much you have learnt :).
Anyway, I don't really have good advice :/, just wanted to confirm that what you feel is expected. Good luck!
Learn how to debug. For me, it's a lifesaver for me to be able to step through some code to figure out what it actually does, instead of me trying to read the code to figure out what it may do. Yes, I do this for my own code too, because we tend to make assumptions, and I want to confirm those, always.
That means learning how to setup your IDE of choice - I presume you use vscode, so you'll then have to google for "vscode debugging". Maybe you'll have to install some addons to add the support, probably setup some launch.json in a local .vscode folder. It all depends on your language of choice.
Learn how to test. This goes great with debugging. I write code in Python, so I end up creating src/ and tests/ folders. src/ for my actual code, and tests/ for my tests. I can use either pytest on the terminal, or just the vscode test addons to run tests.
Anyway, my tests end up being something like this:
src/my_app/main.py or something, with src/my_app/__init__.py existing to turn that folder into a module:
def main():
# some code I want to run
Then in tests/test_main.py (mirroring the src/ folder; adding test_ makes the file findable for pytest, and I call it main to make it easier to link to the main code):
from my_app import main
def test_main():
main()
This is how I always start off with - just a simple piece of code that does a thing, and a test with almost the same name as the function I'm trying to test. I can now add a breakpoint inside test_main and run the test within vscode, which means I have a way of hooking into the main function.
Think about the process of your application
Think about how to cut up the steps to create your application into smaller and smaller steps. Whenever something feels insurmountable, I'll just have to stop in my tracks and mentally cut up a task into smaller and smaller steps, until I feel comfortable to take some steps.
I'm a data engineer, which means I tend to write code to 'ingest' data (which means, grab it from source A and put it into target B (where B is some centralized location to store all raw data).
So the main task is:
Write ingestion
I then have to figure out "what is the source", because that dictates how I grab the data (do I have to loop over all folders in an SFTP server? Is there a state file that makes my life easier? Do I use an API instead?)
Write ingestion
figure out what the source is
Is there an SFTP state file? is it an API?
Do I need a username/password? Some API key?
I then start writing a small piece of code that connects to the source, or just grabs some random data to show the connection works.
So now I can grab some data. Is that data too large to ingest all at once? If a file is super large, I may not be able to hold it into data, which means using a buffer. And how many files are there to download? Should I batch those?
Write ingestion
figure out what the source is | SFTP
Is there an SFTP state file? is it an API? | there is a state file
Do I need a username/password? Some API key? | usename/password
How big are the files?
How many files are there?
and this is how I slowly grow my applications from an idea "ingest all data from some source" into something that can actually run.
Now, I do have some experience and know that filesize and filecount are important to take into account, but that's something I learned along the way.
My first applications just loaded whole files into memory (a bad idea if your memory limit is 4 GB, and I'm trying to load multiple 1GB sized files into memory 😆), and taking local state (which files have I already downloaded) and external state (which one have updated or been added?) into account, etc.
Anyway, you're already on the right path: You already know a weak point, and you're smart enough to know your limits and ask for help when you're stuck. That's one of the fastest ways to grow as a programmer.
I'll often cludge something together just to make it work but I don't feel like I made any progress
That's a good first step! I've been programming for ~25 years and that's still usually where I start. Get a little code that compiles and produces some kind of output or tracing. Then compare the output to your requirements and tweak the code to get it closer to the right behavior. Run it and repeat till it's doing what you want. Do this cycle with small changes, like a handful of lines or a short function, not 20 mins of coding at a time.
Test-driven development can also help with breaking down tasks. It takes a good amount of practice to learn the right patterns, but it's an approach that forces you to work with small narrowly scoped tasks. Then you chain those testable tasks together to create more complex behaviors to create robust testable code.
Experience takes time. Junior developers frequently ask me after I've helped them "but how did you just know how to do that? I've been trying to solve that for an hour and you did it in 10 seconds!!" The answer is because I've solved that exact problem before. More than a few times.
If it's your own code, you can add docstrings comments to your functions, so you don't have to re-read the function body everytime. Also, name functions to be understandable more easily when possible.
If not your code, write on a piece of paper (not on computer) the in and out of a function, maybe like so:
[1,2,3] -> (sum function) -> 6
Then, you can even connect the functions together and see the whole algorithm:
When projects get more complex, paper will not cut it, then some note taking app of some sort will help.
(logseq could help, but some mind mapping or sequence diagram programs would help as well)
Also, I don't know what language your are working with, but learning LISP (maybe clojure) could help.
Why? Because you have to connect your functions together, and it forces you to do so.
At first, it might be harder compared to what you're used to, but it'll give you better fundations to keep learing.
The name of the function, what goes in and what goes out in most cases should be enough to get a good idea on what the function does.
It also helps to make a diagram of how everything ties together. Just boxes and arrows is enough.
When writing your own code, it takes a bit of experience to know when to put something in its own function. It's very obvious when you're replicating code. It's also very common to cut things up when a function gets too big. Look for bits of functionality that you can give a good name.
You might be trying to hold too many details in your head at once. Once you've read a function, try to reframe it in a simpler and easier way; for example, instead of remembering all the steps it makes, just remember what goes in and what comes out (and any side effects). This will let you think about how different functions interact, not just different lines of code.
Write it down when you figure something out. Draw arrows to it from some other part you figured out. Scratch it out when you realize you had it wrong and then put the arrows to a new place where you doodle how it actually works. Never look at the notes again.
Is this with your own code or someone else's? It's always harder to understand someone else's code (at least at first). Everyone thinks and writes in a different way.
In either case, I think you could benefit from stepping through the code in a debugger. Depending on what the code is, give some data as input where you know/can guess what the output is. Using the debugger, step through each line to see what happens to the data. It can help break down long or complicated functions into simpler chunks.
Recognition of functions or snippets of code will come through repetition and exposure. Writing code helps more than reading as well. Even with all of that, it's still okay and common to have to look things up or review. I constantly have to check the syntax of C++ library functions, like snprintf, which I have used but not enough to memorize (and that's okay). Don't be discouraged. I've been in my career for 11 years now, around 9 of which is working with embedded C++, and I still feel like an imposter.
There is a great similarity between watching the variables in a function and juggling. Psychologists will tell you, 'you only get seven variables', but yiu're a coder, you might have ten to twelve( especially if some are boolean), but it's finite, so chunk. Interestingly learning to juggle may help, mirror neurons go brrrr., also chess with moves ahead. If it works it's not stupid, but is it repeatable, testable...
The main part you need to pick up is being able to establish the mental hooks around the ideas that are central to programming. Do you know how you can watch a choreography session and see the dancers just pick up the moves as they’re described/demonstrated? That’s because they’ve learned the language of dance. It’s an entire (physical) vocabulary. It’s the semantics of dance.
What you need to do is do that with programming. There’s a number of getting started with books and videos, but you’re going to want them to learn the fundamentals of not just a language but of programming.
If you’re talking about using other people’s functions (like in an api), then the function name should give you a clue about what it does. The cool thing about functions is that you don’t have to know how they’re doing their thing, just what they’re doing. If you have the source code, you will find you remember more if you use comments to make notes for yourself (it engages more of your brain than just reading).
If your problem is writing your own code using functions, start out more slowly. Write a program that’s just a giant block of linear code. Once that’s working, then take a look as to how to break it down into functions. If you have a block of code that sorts a list, for example, and you had to copy and paste it into three different areas, that would mean it should be a function.
Use comments very often as you’re going. Before you write a block, write a comment about what it’s supposed to do. You’ll start to see some generalities, which will be you learning programming, not just a language.
Difficult concepts are made of simple parts. Find the parts. Find ways to play with them.
Example: TCP/IP performance is complicated, but you can look at specific parts of it like packet latency, retransmission behavior, and the various timeouts. Eventually you understand the "gears" of the system well enough that it's obvious that if the minimum retransmission timeout is 100msec, that a single packet loss means your whole transaction cannot possibly have <100msec latency.
That is exactly what I'm struggling with the most. When it gets broken down like the way you explained it, I can sort of wrap my head around it. But if nobody is there to do that for me I end up confusing myself