Protons and neutrons are held together in a nucleus of an atom by the strong force. The strong force gets it name by being the strongest attractive force. It is 137 times more powerful than electromagnetic, which by the way cannot hold neutrons to protons because neutrons are not charged. It is 100,000 times more powerful than the weak force and 6,000 billion billion billion billion (6 followed by 39 zeroes) times more powerful than gravity which by the way has almost no effect at atomic scales.
According to the standard model of particle physics, the fundamental forces (strong, weak, electromagnetic and gravity) are predicted to occur as a result of an exchange between particles via "force carrying particles". Also, neutrons and protons are made up of tinier particles called quarks. And it is the quarks that exchange force carrying particles between each other to give rise to the strong force. The force carrying particles are called gluons.
It should be mentioned that the strong force only operates at EXTREMELY small distances. These distances are on the order of a 1000th millionth millionth of a meter (10 to the power of -15). If you think about a micrometer (one hundredth the size of a human hair), it is a billion times smaller than that.
The strong force also attracts protons to protons or neutrons to neutrons. In the case of protons to protons, the strong force loses strength after the distance mentioned above and succumbs to the electromagnetic force which pushes the protons apart. In this case the force carrier of electromagnetics is the photon (constituent of light).
So in the nucleus there is a delicate balance of the strong force pulling the atoms in to each other and the electromagnetic force which pushes protons apart. It is only when they are so close together does the attractive strong force overpower the electrostatic.
Protons and neutrons are not fundamental particles like electrons are. That is, protons and neutrons are composed of even more fundamental entities.
Paul Speziale, B.S., Engineering Physics Grad Student, McMaster U.
For this case, protons and neutrons belong to a group of particles called hadrons, which consists of particles made up of quarks. Quarks are particles that can be thought of to be the most fundamental ones like electrons.
The composition of the proton is a combination of 3 quarks -- 2 up and 1 down quarks. For the neutron, the combination is 1 up and 2 down quarks. ("up", "down" and other types of quarks are but fanciful names to distinguish between the 6 different "flavours" of quarks with no literal meaning of the word used. In fact, the term "flavour" -- which is something like "type" in quark terminology -- is also rather fanciful. You don't expect "up" to be a "flavour" in everyday language anyway!)
Quarks interact via the colour (or strong) force, by the exchange of the colour force carriers gluons. In short, quarks are attracted to each other and held together (in certain allowed combinations) by the "colour force" or "strong force".
How, then, do protons and neutrons hold each other together? This can be described as a "leakage" or "residue" of the colour force between the quarks of the proton and the quarks of the neutron that pulls the proton and the neutron together.
This residual colour force therefore manifests itself as the strong nuclear force that binds "nucleons" -- a collective term for protons and neutrons in the nucleus -- together.
This concept might be better understood if we understand that -- by analogy -- atoms, being electrically neutral, should not be attracted to other atoms to form molecules. However, due to the composition of the atom (positive nucleus and outer negative electron cloud), atoms do come together and form molecules. In an analogous way, we can understand the strong nuclear force by understanding the composition of the nucleons first.
There are other ways to answer this question, and one of them is the concept of binding energy.
One of the tendencies of nature is to achieve stability. And a way to go about doing it is by minimizing the energy of a system.
When protons and neutrons "combine" to form a nucleus, we notice that the mass of the nucleus is less than the sum of the masses of the constituent nucleons. This is known as the "mass defect". How do we explain that?
Mass and energy are equivalent, and their relationship is immortalized in that famous equation E = mc2. Essentially, it means that mass and energy are different forms of the same stuff (just like potential energy and kinetic energy are but different types of the same thing called energy).
The mass defect is simply the amount of mass (or energy) that is released when protons and neutrons come together to form a nucleus. Generally, the greater the amount of mass defect per nucleon, the more stable the nucleus.
Hence, the holding together of protons and neutrons in a nucleus can also be explained by the concept of binding energy and mass defect.
All these explanations must explain that the force of attraction is larger than the repulsion between positively charged protons, over the range of the nucleus that is.
Ryan Leong, Undergraduate, NUS, Singapore