The Origin of the Misconception
Samurai vs ninja, the myth of the ninja being an oppressed underclass has been central to the image of the shadow warrior since its introduction in the West. Even today there is debate about the two groups. Some say that the ninja were separate from the samurai, while some have tried to say that the ninja were no more than samurai with a special job. As with most things from a culture as different from the West as Japan's, the truth is a bit more layered.
To explain the matter, we really need to look at the whole of the Japanese military situation and the classes that existed within it. People tend to lump all periods and areas of Japan as one, even though the area and passage of time actually covers more than from the Norman invasion of England to the condottieri of renaissance Italy. What is true of 12th century Japan has very little to do with the peace of the 18th century Edo period. Yet terms such as samurai are used in both eras. Just as there are changes in what a fighting man in Europe was, and much was not really covered as much as other subjects, so too Japan has a lot of things that are true if you are talking about one time, but not in a later age. A lot of what is thought of in the West tends to have been twisted by misconception when translations have been made.
Every nation and culture has had men that went out and fought. The same can be said for spies and those that did related jobs. Every ruling elite class has tended to write more about themselves than those that served in lesser roles on the battlefield. While the bulk of an army might have been peasant conscripts or those that join, they garner little mention in records and it usually takes careful research to ferret out their true nature and role. In addition, every era seems to look on prior ones with admiration and this goes double for warriors in eras of peace looking back on their more experienced ancestors.
Japan's warrior traditions went through centuries of evolution just as every other nation. To tell the full story to satisfaction, we must start in the early history and work forward.
Prehistoric Japan, that is to say before written records were kept, is obviously a bit murky and subject to endless debates. The relationship between Korea and Japan is especially contentious and tinged with nationalist sentiment on both sides. There obviously was some relationship with the very early forces that came to rule all of Japan and one of the nations that existed in Korea. Korean historians like to paint this as a case of Korea colonizing Japan, while Japanese historians take the stance that native Japanese formed ties to the mainland in order to benefit from trade and the advances they could gain in technology and culture.
In any case, it looks like some with horses and metal technology made their way to Japan and helped form what is known as the Yamato court. The name is taken from the area, not far from Kyoto, Nara and Iga, that early governments that led to the current imperial line were set up in. Until the founding of Nara in 710, the court tended to move with each succeeding ruler.
The idea for a permanent capital came from the Chinese, with which the Japanese were in diplomatic contact. By this time in history Buddhism, writing systems and technologies such as rice farming had been established in Japan from the mainland. The Imperial Court ordered that official histories be made based on the oral tales and such that had been passed down until then. Aside from the problems of basing things on the vagaries of stories passed through several tellers, the imperial court sought to establish their power by modifying the tales to reflect the primacy of their rule. In Europe, kings justified their rule by the idea of divine right that said that those in power had to be there by the will of god. An unexpected result of this by those that created it was that anyone who deposed the current monarchy and set themselves up in their place must have god's blessing to do so. China's divine mandate had much the same effect. In Japan, the official histories like the Kojiki and the Nihonshoki set up the sun goddess Amaterasu as supreme deity and the imperial line as her descendants. Thus, no one deposing them by force would be legitimate.
Histories (to use the term loosely in this case) such as the Kojiki are scant on accounts that would give us insight into the way wars were fought and the men that participated in them. No accounts of how battles were waged are given, mainly only the results. The same can be said for the gear or structure of the warriors on the battlefield. Obviously men fought, but we can't even find many references to cavalry, much less what kind of social class served in them. It is believed that the elites served as a mounted force while lower classes were pressed into service as infantry. This is just the way almost every culture has structures their forces. In addition, the keyhole- shaped burial mounds for the upper class known as kofun from this time were filled with clay figures called haniwa of military men as well as those of horses with saddles.
The stories do tell us some things. For one, the story of imperial prince Yamato Takeru is interesting in that he engaged in some acts that we would consider very ninja- like. In one case, he dressed as a female and allowed himself to be lured into a rival's embrace, where he then assassinated him. In another, he had a wooden sword made and suggested that he and the other he was in negotiation with exchange swords. Obviously neither trusted each other and each put down their sword in unison and moved to where the other dropped their sword. This was to keep one from grabbing both swords and putting the other to death. Little good it did the rival when he found that he had been duped and ended up killed by his own sword.
Some Japanese scholars look on this as some of the first examples of ninjutsu in Japan. The truth is, this type of trickery is no stranger to any culture. There is no indication that any of these feats had any link to later ninja, but that did not stop a lot of the ninja in the peaceful Edo period from trying to build themselves up by forging a link to an ancient imperial prince.
More interesting is what the underlying messages of these stories indicate. An imperial prince, one held up as a standard for honor, using deceit and assassination. This kind of casts a new light on the myth that the ninja did jobs that the samurai could not stoop to for fear of dishonor.
Once the first permanent capital was built at Nara in 710, to change location to Kyoto by the end of the century, the centralization of power increased dramatically. Actually, the trend had been going on for well over a century and when Shotoku Taishi had been made regent in 593 it was clear that the imperial family had given up on being warriors and had delegated that role to other parts of the government. At first clans such as Mononobe served as the warriors, but those clans had their own agendas and the court moved to make military forces directly loyal to them. Part of this was the rise of a centralized army. Peasants were conscripted for a period and many were sent to garrison bases opposite of Korea. There was a real fear of an invasion, one that passed in time. But for those who were conscripted, it was a disaster for their families. Eventually, the system just broke down as the threat of invasion from Korea faded over time. In fact, the entire government system rotted as regional governors and such carved out their own power bases at the expense of the central government. As money was diverted to local powers and military might controlled by the central government declined, vast areas of Japan became ungovernable. While they were loyal to the emperor on paper, they in fact were not under any control by the court.