Btukfyl

For example, RSA relies on mathematically hard problem, factoring, while ECDSA or similar rely on "discrete logarithm problem".

What makes SHA256 and similar hash functions, of the same family, secure against pre-image and collission attacks? Whats the math behind it?

The design of SHA-256 relies on two structures, one-way compression function which is based on Davies–Meyer structure and Merkle–Damgård structure that uses this Davies–Meyer structure.

• Compression function: transforms $2n$-bit input into $n$-bit. The transformation performed in a way that it achieves avalanche effect, i.e. every output bit depends on every input bit.




• One-way function: Easy to compute hard to invert.



• 
  

  One way compression function should have these properties;

  
  
  
  
  
  1. 
     Easy to compute: the calculation of the output is easy for a given input.
  
  
  2. 
     Pre-image resistant: given a hash value $h$ find a message $m$ such that $h=Hash(m)$. Consider storing the hashes of passwords on the server. Eg. an attacker will try to find a valid password to your account.
  
  
  3. 
     Second Pre-image resistant: given a message $m_1$ is should be computationally infeasible to find another message $m_2$ such that $m_1 neq m_2$ and $Hash(m_1)=Hash(m_2)$. Producing a forgery of a given message.
  
  
  4. 
     Collision resistance : if it is hard to find two inputs that hash to the same output $a$ and $b$ such that $H(a)= H(b)$, $a neq b.$
     
  
  
  
  

(SHA256 Compression function, from Wikipedia )

• Davies–Meyer structure is a one-way compression function based on a block cipher. Security of this construction in the Ideal Cipher Model. However, there is a property of this construction; even the underlying block cipher is secure it is possible to find fixed points.




• Merkle–Damgård structure (MD) uses a compression function. MD is collision resistant if the compression function is collision resistant one-way compression function.. MD constructions have length extension attack that SHA-256 is also prone to this attack.

Note: There is a preimage resistance attack for 52 out of 64 rounds of SHA-256.

It's worth pointing out that in the case of SHA2 and most other hashes the compression function has a block cipher (keyed permutation) as its core.

Basically what you are asking is identical to asking how can block ciphers be resistant to known-plaintext attacks and chosen-plaintext attacks (arguably doesn't apply to SHA2 specifically because an attacker doesn't control that aspect) and even related-key attacks in the case of SHA2 (because it uses a Davies-Meyer construction where the attacker has control over what gets fed into the key schedule).

There is no proof that this methodology is reducible to something that is proven secure. It is believed to be secure due to diffusion and confusion properties which as far as is known allow no efficient backtracking. You can think of it as extreme sensitivity-to-initial-conditions in a discrete non-continuous domain.

Edit: The reason I went to block ciphers is because hash security is provably reducible to the security of the core keyed permutation (or even unkeyed if you look at SHA3) - that's how hashes are designed to begin with. Which I believe is the spirit of your inquiry. But the buck stops there, no security proof for those exists.