Isn't it kind of their job to determine if something violates the constitution though?
That was a power the court took for themselves. The court didn't start out that way and many consider the power grab to be unconstitutional.
Yes, many people do that. But how do they square that opinion with the plain text of the Constitution?
"This Constitution, and the Laws of the United States which shall be made in Pursuance thereof; and all Treaties made, or which shall be made, under the Authority of the United States, shall be the supreme Law of the Land"
If the SCOTUS has no power of judicial review then in what sense are the Constitution and the laws made in pursuance thereof the supreme law of the land? If Congress can do whatever it pleases and nobody can overrule them then every random federal law not in pursuance of the Constitution is the supreme law of the land.