On Problem Solving

I have wasted a lot of time wondering how to approach a problem or study a subject. Well, it wasn't all a waste, though! Some things have indeed stayed with me and have changed my way of working.

But nothing has influenced me more than recognizing the temperament that aids in doing profound work: the courage to dive into dirty ditches and wrestle, the curiosity to seek a broader perspective, and the capacity to concentrate alone.

Mathematical Thinking

Mathematical thinking involves the skills of modeling a given problem formally and rigorously arguing about the formal statement.

Here, the point of rigor is not to destroy intuition, but to eliminate bad intuition while clarifying the better.
Even though rigor is merely the instrument of demonstration, just as intuition and curiosity are the instruments of invention, there is no denying that it alone can provide us with certainty.

Moreover, in mathematics, we often find two different cultures involved. There's truth in both cultural approaches and it's even not the case that people always wholly subscribe to one. The categorization might therefore be an oversimplification but it is useful nonetheless.

Culture A: The point of understanding mathematics is to become better able to solve problems.
Culture B: The point of solving problems is to understand mathematics better. Obtaining a solution is just a step, the goal is to increase your understanding of a subject.

Algorithmic Thinking

Algorithmic thinking is quite similar too, except a few major variations. These differences involve the constant juxtaposition of some central notions inherent in the art and science of informatics. Like:

notion of complexity and economy of operation (complexity theories)
dynamic notion of the state of any process (data structures)

These notions are naturally associated with the idea of implementability, which can be used to put forward the following picture — is computer science implementable mathematics? And if it were so, then computer science is rooted in physics as much as we believe it is rooted in mathematics. After all, it is the nature of our universe and existence which determines what is implementable and what is not.

Meta-thinking

Studying and working through problems often seem to depend on implementing your reasonings. While implementation might be a matter of grinding and hard to categorize, reasoning can be simplified into types:

deductive - proofs, formalism
inductive - abstractive, bayesian
abductive - randomness, speculative questioning
analogical - drawing parallels

Furthermore, the ability to reason well seems to depend on knowledge (nodes), understanding (representation of these nodes) and extrapolation (guided random search, open-ness, active discovery).

Heuristics

Proofs are Programs: Often, we find that the approaches we make (both mathematical and psychological) while trying to prove some statement or construct an algorithm are similar.

This becomes clearer when we start reading more about the correspondence between proofs, programs, and algebraic structures. Such a trinity allows us to say with certainty that proofs and algorithms are complementary and analogous to a great extent. Furthermore, thinking about proofs and programs in this manner often gives us a clearer perspective while approaching any given problem.
Abstraction and Modularity: Create good abstractions such that you're able to compress the problem space enough to fit it in your head. Naming your concepts help. Modularizing your problem also helps in reducing complexity.

You can start off by dividing you problem in layers of chunks and while solving for a specific chunk, consider everything else as a blackbox. You can further hope to simplify each chunk or a set of chunks by reducing it to another problem or wishing off several constraints and trying your hand at that. When dealing with implementation jobs, then this methodology is also referred to as the top-down approach, albeit with some more caveats sprinkled all-over. Similarly, while constructing any proof, you ought to try stringing lemmas together, along with speculation that has been cleaned and sharpened with rigor.

Another way to reduce complexity is to abstract out the problem and deal with fundamental assumptions. Ab initio cum recta abstractio.

"Suppose that you are given a problem to solve, I don’t care what kind of a problem - a machine to design, or a physical theory to develop, or a mathematical theorem to prove, or something of that kind - probably a very powerful approach to this is to attempt to eliminate everything from the problem except the essentials; that is, cut it down to size. Almost every problem that you come across is befuddled with all kinds of extraneous data of one sort or another; and if you can bring this problem down into the main issues, you can see more clearly what you’re trying to do and perhaps find a solution. Now, in so doing, you may have stripped away the problem that you’re after. You may have simplified it to a point that it doesn’t even resemble the problem that you started with; but very often if you can solve this simple problem, you can add refinements to the solution of this until you get back to the solution of the one you started with." (Claude Shannon)
Perspective and questioning: When solving a problem it is vital to keep changing your perspective. You need to look at it and attack it differently (using different tools with different tricks). For example, try contradictions, then maybe induction, then maybe something else and so on. You also need to be aware of how the problem and its possible solution might generalize or fit into the larger scheme of things.

"While looking at a problem through different lenses ideally you would want to find the natural world for the problem, express it cohomologically and often the cohomology of that world may solve your problem, like a ripe avocado bursts in your hand." (Grothendieck)

Likewise, never hesitate to keep asking questions, no matter how dumb they might seem. Don't take anything you don't really understand for granted! Where's the fun in that? Try sample cases and examples if needed. Try to solve for special cases if present.

Also, you need to realize how crucial partial progress is. Failures are often crucial advances. In fact, people should write about their failures and speculations while writing a paper. Because well, if we don't at all have any insight into what made someone come up with an argument or construction, then what is the fricking point at all?

"Change the words. Change the viewpoint. Look at it from every possible angle. After you’ve done that, you can try to look at it from several angles at the same time and perhaps you can get an insight into the real basic issues of the problem, so that you can correlate the important factors and come out with the solution. It’s difficult really to do this, but it is important that you do. If you don’t, it is very easy to get into ruts of mental thinking. You start with a problem here and you go around a circle here and if you could only get over to this point, perhaps you would see your way clear; but you can’t break loose from certain mental blocks which are holding you in certain ways of looking at a problem. That is the reason why very frequently someone who is quite green to a problem will sometimes come in and look at it and find the solution like that, while you have been laboring for months over it. You’ve got set into some ruts here of mental thinking and someone else comes in and sees it from a fresh viewpoint." (Claude Shannon)
Formalism and Writing: Formalism is arguably the most important and recurring step towards solving any problem. This is something I learned while studying functional programming. Often, the act of describing a problem and stating it formally reveals what you need to solve it.

And in case you are dealing with an algorithm or a computer system, formalism through programming is both the crux and beauty of it. Similarly, on a broader note, writing too is hugely essential. Ideally, it should be the primary mechanism for doing research and not just for reporting it.

References

The Two Cultures of Mathematics by Gowers
On Proof and Progress in Mathematics by Thurston
Poincaré on Intuition in Mathematics
Algorithmic Thinking and Mathematical Thinking by Knuth
There’s more to mathematics than rigor and proofs by Tao
Ask yourself dumb questions by Tao
Solving mathematical problems by Tao
How to write a great research paper
Seekers and Craftspeople by Lee Smolin
Birds and Frogs by Freeman Dyson
Some blog on codeforces
Thirty years of proof: an interview with Andrew Wiles