The common explanation for how otherwise reasonable people end up believing in conspiracy theories is straight-forward and easy to empathize with, only having the fault of being completely wrong. According to common wisdom, some people, when faced with a shocking or upsetting act of violence (e.g. the Kennedy assassination, 9/11, any mass shooting), are unable to accept the chaotic violence of the world and buy into a conspiracy theory as a coping mechanism. After all, isn’t it more comforting to pin all the blame on some shadowy organization that we can fight? This is comforting for us non-conspiracy theorists, with the added bonus of being subtly patronizing towards CTs (i.e. conspiracy theorists. I don't feel like typing it a hundred times). This explanation means that, deep down, they realize the same chaos we do, but they’re just really bad at coping, the poor dears. That it doesn’t make sense has been no obstacle to it becoming commonly accepted. The argument that CTs find this conspiracy-haunted world comforting is identical to the claim they make about those of us who don't buy in to their grand conspiracies, that it must be nice to live in a world where individuals are responsible for the bad things they do, and we can trust our institutions, etc. Further, our explanation absolutely fails to address things like, say, 9/11 truthers. 9/11 wasn't a lone wolf assassinating a president or some freak accident. What emotional need is satisfied by shifting the blame from Al-Qaeda to the Illuminati/New World Order/etc.? It's much easier to fight Al-Qaeda than a shadowy secret organization that nobody believes to exist.
The answer to how people come to believe in conspiracy theories is pretty simple: The same way as they come to believe in anything else. Let's take the Kurt Cobain murder conspiracies as an example. When a person is faced with new information, that information must be assimilated. If that information fits in with the things you already know, it's a smooth process. (N.B. I'm using the words "know" and "knowledge" to mean statements that one holds to be true, regardless of their factual accuracy.) This is why there aren't as many conspiracy theories about the deaths of Jimi Hendrix or Amy Winehouse. Even their fans know that they both did drugs and that drugs can have tragic consequences. In regards to Cobain, one of the frequent claims made by those who believe he was murdered is that he wasn't suicidal. They know how suicidal people act, and they know that Cobain wasn't acting that way before his death. When this knowledge is contrasted with the knowledge that Cobain died in the way he did, some piece of knowledge must change so the new information can be assimilated. Maybe suicidal people act differently than I thought they did? Alternatively, the new information must be denied. Cobain didn't commit suicide. A more recent theory is that Cobain faked his death, though the prominent theory is the Courtney Love had him killed. But how do we get from "Kurt didn't seem suicidal" to "Courtney Love had Kurt killed?" The steps are pretty easy to trace. If Kurt didn't commit suicide, then he must have been murdered. One of the earliest theorists was a public access host who got footage from a window outside the crime scene and noticed much less blood than he would expect from a shotgun blast to the head. The host, and those who agreed with him, knew how much blood to expect. Also, if one accepts that Cobain was murdered, someone else must have written the suicide note. After all, the note doesn't look right. Which is to say, it doesn't look the way you'd expect it to. And everyone knew Courtney Love was only in it for the money and didn't really care about Kurt.
My point here is that conspiracy theories and theorists don't start with a full-fledged master plan. To give one more example, consider the various Shakespeare authorship theories. Typically, people just write them off as snobbery, but the reason is a bit more complex. It also shows that the types of things we "know" aren't always as concrete as in the Cobain conspiracy. The lack of contemporary documentation about Shakespeare isn't unusual, as even many Oxfordians, Baconians, and Marlovians will acknowledge. The real problem is that what we do have shows Shakespeare to be, well, boring and occasionally unpleasant. As James Shapiro shows in Contested Will, it was not Shakespeare's class that dismayed early CTs, but the fact that the few things we know about him show his stinginess (e.g. suing a neighbor over a small debt), and that he retired to his estate in Stratford rather than continue writing in the last six years of his life was unacceptable. Everyone knows that the greatest literature in the English language must have been written by an equally great spirit. Even we who accept Shakespeare of Stratford as the author of his plays feel this disappointment, yet it is a disappointment based on our assumptions about the way the world works. Most of the supporting evidence for these conspiracies are based on accepting certain unproven premises (e.g. the sonnets are autobiographical) which can be construed to prove nearly anything, or by misapprehensions (e.g., there are no records that Shakespeare went to grammar school, therefore he did not. However, there are no grammar school records of any of Shakespeare's peers, some of whom went on to attend Oxford). But the reason there are conspiracies about Shakespeare at all is the belief we have about the personal qualities of a great artist.
We can see in the given examples how conspiracy theories arise logically. Logically, if not reasonably. They are logical in the sense that they take a set of given conditions (this is how suicidal people act, this is what a great artist behaves like), and then apply logic. If he didn't kill himself, who wrote the suicide note? If vaccines cause autism, why are doctors saying they don't? If these people standing outside a mass shooting aren't behaving how survivors behave, then who are they? To bring it back to my initial point, what's comforting about believing that your favorite musician's killer is still at large, or that the government is dispersing toxic chemicals from airplane jets?
Conspiracy theories, by their nature, have a tendency to broaden their scope. Let's say you start with the knowledge that the US intelligence agencies are close to omniscient and that middle eastern extremist groups are unsophisticated, and then came to the conclusion that 9/11 was an inside job. On the one hand, this would need to implicate a lot of people not directly associated. If you 'know' how a building would appear when it collapses, and countless structural engineers say differently, then they must be lying. On the other hand, this would tie into other conspiracy theories already held. If you believe that there is a Jewish plot to control the world, you'd find a way to tie 9/11 to that.
Understanding why people believe in conspiracy theories, and how those beliefs develop is important. I hope it's already understood that most conspiracy theorists are not violent or bigots. But as online communities continue to supplant physical ones, we have to take a few things into consideration. First, groups like Stormfront (the white supremacist forum) have a history of trying to recruit from places like reddit's conspiracy page, not because the groups targeted for recruitment are necessarily racist, but because if (as the neo-nazis believe) there are shadowy Jewish groups controlling everything, why not try to make your case to people who already believe there is a shadowy group pulling the strings? Second, many of the views espoused by conspiracy theorists are socially unacceptable. (In the cases where they accuse innocent people, entire ethnic groups, or survivors of tragedies of unspeakable crimes, I'd say this unpopularity is justified.) Online communities are still communities, and fulfill that need. Communities form identities and protect themselves from perceived threats, even if that means protecting bad behavior within a community. Third, there is still a general feeling that online is not real, not just in terms of community but in terms of actions. As such, harassment of people accused of complicity in a conspiracy has become a serious problem, since there are so rarely any actual consequences for the harassers, even when their activities are clearly illegal.
There have always been and will always be conspiracy theories. What needs to be discussed is not how to stop them overall, but how prevent harm to innocent people caught up in them, whether we're talking about the anti-semitic results of "The Protocols of the Elders of Zion," the careers destroyed by McCarthyism, or the harassment of bereaved parents after a school shooting. On the part of conspiracy communities, the answer is better self-policing, primarily in regards to preventing co-opting by hate groups and, as a community, establishing a clear sense of opprobium for harrassment. On the part of broader society, we need to take online actions seriously. Criminal harrassment and death threats must not be consequence free simply because they are carried out online.