Stalin was paranoid, that much is clear. The question is what was he hoping to accomplish and who he thought his enemies were. I've read much speculation on the subject, and speculation it must remain, as he was not open about his reasons. Some people suspect that the purges he launched late in his life were directed against Lavrenty Beria, the head of the NKVD. They failed because Beria had too tight a control over the security apparatus. (Kruschev was wiser, when he needed to take down Beria, he enlisted the one organization powerful enough to do it, the Army. Kruschev had spent years cultivating friendships with the upper leaders of the Army, and many of these same leaders feared purging by Beria, so it was easy for him to arrange the coup). Many still speculate that Beria had Stalin poisoned, and then killed the doctors to cover his tracks. Of course, there is no evidence.
As far as actual crimes to be punished, I think it is safe to conclude the victims of the purges were no more guilty than average citizens arrested at random would have been. In 'the Gulag Archipellego," Solzenitzen describes the processes of the justice system during this time. Lack of evidence was no object. In one memorable example, at a trial, both the prosecutor and the defense attorney recommended the charges against the defendant be dismissed, but the judge went ahead and convicted him anyway. Ordinary citizens were not even allowed to read the laws of the country. There were laws in place to protect them, but most were unaware, and the authorities used this to their advantage.
However, Stalin had so many purges, there's no reason to think that they were all for the same purpose (except, in a broader sense, to eliminate rivals and so forth). By the time the great officer purge of 1940 came around, he had already eliminated all his significant political rivals. Possibly he feared that the upper leadership of the army was too aligned with their former commander, Trotsky, but there's no real evidence of that.
Unlike Mao and his Great Proletarian Cultural Revolution, which was really just a purge disguised as a cultural movement, there was no obvious target for Stalin's purges. (the targets for the Cultural Revolution were Liu Shaoshi and Lin Biao)
I suspect that the purge of 1940 was largely an attempt by Beria to strengthen the NKVD by removing those who wanted to strengthen the military; particularly the new-dangled airforce. Of course the outbreak of war, and the astonishing success of the German invaders led to a redoubling of efforts - after all, if Russia is great, and her leader is great, then only sabotage at the highest level can explain their failure.
The real failure was Stalin's, of course; it seems that the only person that the intensely paranoid Stalin ever really trusted was Adolf Hitler; Stalin really wanted the Molitov-Ribbentrop pact to hold, and even after the Germans struck deep into Russia, telling him that it had not (with the implication that Hitler had hoodwinked him) was a life threatening move for his top military commanders.
It was never a good plan to tell Stalin things he didn't want to hear; and their being true was often not a defence.
Acts of random and extreme violence and barbarity are a very effective way of consolidating one's grip on power; if everyone is crapping themselves that they might be next, you can be assured that nobody will risk crossing you. But this comes at the price of nobody daring to tell you things you need to know, but don't want to hear. And of course, you must be completely pitiless.
Stalin, Hitler, Mao and Pot were pitiless; they really did view the deaths of millions as a mere statistic.
Their successors foundered, because they were humane. Gorbachev refused to crush the Polish 'Solidarity' led uprising (both literally and figuratively) with Russian tanks; and so the whole thing unravelled. Truly merciless leaders can only reach the top by fighting their way there. Once the revolutionaries of the old guard were gone, their successors were never going to have the stomach for the level of madness needed to keep the lid on; particularly over the wider Warsaw Pact.
People, at every level, seem to have become far less inclined to use massive and indiscriminate violence over the last half-century or so. That makes full-on dictatorship much more difficult to sustain.
Not long ago, killing a million or two peasants was almost unremarkable. Today, you massacre a mere few dozen, and everyone loses their minds.
I blame the UN, TV, and the rise of rapid global mass media. People used to want the truth, but they couldn't handle the truth, so governments kindly kept the truth from them.
Now that the truth is out there, the people still can't handle it; so they act as a brake on the worst excesses.
That's probably a very good thing.
you anarchistic running-dog, you. It's a joke.