People want to hire someone who helps them get things done. Lots of companies I work with and for have had problem solving types who can't carry on a conversation without staring at their shoes. This can get you to a certain point. Even a pretty high point on some technical track if you're good. But at some point if you want to make it on the general ladder you need to show you can think a little bigger. At some point you have to make people think you can lead and manage people.
The ability to bullshit is definitely one of the most valuable traits in the modern workplace. If you are honest about your talents and your achievements you will almost certainly do less well than your contemporaries. I often see people who are really rubbish at what they do get promotions or opportunities due to their ability to spin their failure into sounding like success.
I also see extremely talented individuals who add significant worth to the institution they work for (often to the extent that the projects would have failed if not for their input) be passed over for promotion and recognition simply because they are not forceful enough in shouting about their contribution. BTW, I think this is a significant contributor to the gender pay gap.
But in my opinion, the real "problem" is that managers are paid more than technical people. There is a perception that one should be promoted from technical roles into management roles. But why? They require very different skillsets and both are often essential to the functioning of a company, so why not regard them on an equivalent footing? Don't encourage outstanding technical people to apply for "promotion" to management positions they are unsuited for and will probably be unhappy doing just because they will get more money.
On the topic of luck, I would agree with the initial hypothesis that it is important. However, I think for most people, it sort of cancels out. If you have a long career, you have sufficient good luck and bad luck for them to normalise each other. Having said that, the people who are really successful are usually those who have stacked up the luck along the way.