Btukfyl

In security news, I faced a new term related to hash functions: it is reported that the "in-house hash function" used in IOTA platform is broken (i.e. Curl-P hash function). You can find the complete paper introducing the vulnerability here.

But I do not understand if the term of "in-house" represents a specific type of hash function? And in general, what does "in-house" mean here?

From the explanation of in-house in the Cambridge Directory: "Something that is done in-house is done within an organization or business by its employees rather than by other people".

Here it means developing your own hash algorithm instead of using a public one. Usually that means that it is developed by only a few people with only limited expertise in the problem area and without any public input. Thus it is very likely that the self-developed one gets eventually broken once more experts in cryptography take a look at it.

See also Why shouldn't we roll our own? and How valuable is secrecy of an algorithm?.

In the context of cryptography "in-house" is a synonym for "questionable origin and unverified strength".

It specifically means that they developed their own hashing function (or in other cases encryption, key-exchange scheme, etc.).

This, in cryptography, is a Bad Idea with capital letters. While developing your own library of common functions or your own webservice framework or whatever can have a perfectly good use-case, cryptography is one of the fields where a tiny mistake can make the whole thing incredibly fragile in a way that you will never find out. If you build your own webserver ("our high-performance in-house webserver...") and there's a problem, you have a good chance of finding out sooner rather than later because it crashes, or sends the wrong files, or performs badly. But if your crypto algorithm has a problem that destroys its cryptographic strength, you have to be very lucky that someone who breaks it actually tells you. The people who try to break it are almost certain to be attackers, because very few cryptographers waste their time on some in-house crypto hack. They know to stick with public algorithms where it actually matters if they find something, to more than one company.

I agree with the answer given an hour before this one about in-house meaning, "non-standard and probably not very sophisticated or rugged." There may still be one argument in favor of using an in-house hash. That is, it may be different enough from the standard ones out there that a hacker may decide it is too much work to figure out how to reverse engineer it. Even if you accept this argument, this sort of do-it-yourself approach should only ever be used to protect very low-value data.