A Treatise on Thinking and Problem Solving
I have wasted a lot of time metaproblem solving. Or more specifically, wondering about how to think about approaching a problem. Most of these venturing was during my preparation for ICPC (and eventually World Finals). It wasn't all a waste though. Somethings stayed and changed my way of working. Firstly, the differences in approaching problems based on culture (read: particular field) and character (read: seeker or craftspeople). Secondly, the similarities across the board which could be generalized. Cliche, much?
 Do mathematicians, physicists and computer scientists approach a given problem differently? Is mathematical thinking different from algorithmic thinking? I love the word algorithmic rather than computational because the latter presents an ambiguous scope of considering the purely mechanistic working of a computer.
 Are there any generalizable heuristics that can be used or applied to any formal problem?
In this essay, I will try my best to put forward several ideas I have come across and as they have been filtered and modified by my own point of view. References to interesting reads are present at the end.
Mathematical Thinking and Algorithmic Thinking
Mathematical thinking and algorithmic thinking have a lot in common. It shall be easier to first define what we consider mathematical thinking to be and what it means to be a mathematician.
Mathematical Thinking
Mathematical thinking involves the skills of modeling a given problem formally and rigorously arguing about the formal statement.
 Here, the point of rigor is not to destroy all intuition; instead, it should be used to eliminate bad intuition while clarifying and elevating good intuition.
 Even though rigor is merely the instrument of demonstration, just as intuition and curiosity are the instruments of invention, there is no denying that it alone can provide us with certainty.
Moreover, in mathematics, there are two cultures involved. They differ based on the following two ideologies.
 The point of understanding mathematics is to become better able to solve problems.
 The point of solving problems is to understand mathematics better.
I subscribe more towards the second culture, probably because I have a greater passion for theory building. It feels quite undeniable to me that there is something truly romantic about extracting underlying structures and generalizing them to create a much bigger, expressive, and beautiful framework.
Differences
There are two major differences between mathematical and algorithmic thinking. These involve two basic notions inherent to computer science and are totally absent (to a large extent in mathematics).
 notion of complexity and economy of operation (complexity theories)
 dynamic notion of the state of any process (data structures)
These notions are inherently associated with the idea of implementability, which can be used to put forward the following picture — is computer science implementable mathematics? And if it were so, then computer science is rooted in physics as much as we believe it is rooted in mathematics. After all, it is the nature of our universe and existence which determines what is implementable and what is not.
On Heuristics

Proofs are Programs: Often, we find that the approaches we make (both mathematical and psychological) while trying to prove some statement or construct an algorithm are similar.
This becomes clearer when we start reading more about the correspondence between proofs, programs, and algebraic structures. Such a trinity allows us to say with certainty that proofs and algorithms are complementary and analogous to a great extent. Furthermore, thinking about proofs and programs in this manner often gives us a clearer perspective while approaching any given problem.

Abstraction and Modularity: The set of ideas I hope to entail here also go by several other names, like, wishful thinking and thinking from first principles. However, the main idea is following a modular approach while constructing an algorithm or proving a theorem.
You can start off by dividing you problem in layers of chunks and while solving for a specific chunk, consider everything else as a blackbox. You can further hope to simplify each chunk or a set of chunks by reducing it to another problem or wishing off several constraints and trying your hand at that.When dealing with implementation jobs, then this methodology is also referred to as the topdown approach, albeit with some more caveats sprinkled allover.
On a similar note, it is worth remembering that while constructing any mathematical proof, you ought to try stringing lemmas together, along with speculation that has been cleaned and sharpened with rigor.

Questions and Perspective: When solving a problem it is vital to keep changing your perspective. You need to look at it and attack it differently (using different tools with different tricks). For example, try contradictions, then maybe induction, then maybe something else and so on.
While looking at a problem through different lenses ideally you would want to find the natural world for the problem, express it cohomologically and often the cohomology of that world may solve your problem, like a ripe avocado bursts in your hand. (Grothendieck)
Likewise, never hesitate to keep asking questions, no matter how dumb they might seem. Don't take anything you don't really understand for granted! Where's the fun in that?
Also, you need to realize how crucial partial progress is. Failures are often crucial advances. In fact, people should write about their failures and speculations while writing a paper. Because well, if we don't at all have any insight into what made someone come up with an argument or construction, then what is the fricking point at all?

Formalism and Writing: Formalism is arguably the most important and recurring step towards solving any problem. This is something I learned while studying functional programming. Often, the act of describing a problem and stating it formally reveals what you need to solve it.
And in case you are dealing with an algorithm or a computer system, formalism through programming is both the crux and beauty of it. Similarly, on a broader note, writing too is hugely essential. Ideally, it should be the primary mechanism for doing research and not just for reporting it.
Solving problems depend on the ability to reason and implement your reasonings well. The ability to reason well is dependent on your knowledge (nodes), understanding (representation of the known nodes in a graph) and extrapolation (randomness is all you need). Reasoning is generally of three types:
 deductive: proofs, formalism
 inductive: abstractive, bayesian
 abductive: randomness, speculative questioning
 analogical: drawing parallels
In case of competitive programming, for example, reasoning involves making observations and applying techniques.
 Observation: understanding the problem and come up with nontrivial properties.
 Technique: appling known algorithms or data structure to the problem.
 Implementation: writing the solution fast, bugfree and naturally understandable.
In the end, all you need is courage, the strength to back it up and to love the pursuit of the same.
Conclusion
Heuristics, as mentioned above, serve as more of a psychological strategy than a tangible cut out path for solving problems. Almost all major problems that lurk in the horizon of human understanding are mainly solved by experience (of your own field and others), curiosity, hard work and peeking into randomness.
Furthermore, we don't really understand how a human being thinks. We do have some understanding that our linguistic capabilities serve as an operating system for our minds supporting abstraction and other cognitive abilities. But, there is so much that we don't understand. However, do not let this lack of understanding reflect any kind of unimportance on the subject  for if we don't even understand how we think, how could we possibly make machines that can truly think like and for us?
References
 The Two Cultures of Mathematics by Gowers
 On Proof and Progress in Mathematics by Thurston
 Poincaré on Intuition in Mathematics
 Algorithmic Thinking and Mathematical Thinking by Knuth
 There’s more to mathematics than rigor and proofs by Tao
 Ask yourself dumb questions by Tao
 Solving mathematical problems by Tao
 How to write a great research paper
 Seekers and Craftspeople by Lee Smolin
 Birds and Frogs by Freeman Dyson
 Some blog on codeforces