CHAINMAIL (combat rules used for OD&D)
-------------------------------------------------------
"Missiles cannot be fired into a melee... Missile troops interspaced w/ other footmen forming a defensive line may refuse combat & move back 3" out of melee range."
HOLMES BASIC
----------------------
"Once the party is engaged in melee, arrows can not be fired into the fight b/c of the probability of hitting friendly characters."
AD&D DUNGEON MASTERS GUIDE
-------------------------------------------------
"discharge of missiles into an existing melee is easily handled. It is permissible, of course, & the results might not be too incompatible w/ the desires of the discharging party. Assign probabilities to each participant in the melee or target group according to sheer numbers. In the case of participants of varying size use half value for size "S", normal value for size "M", and one & one-half value for size "L" creatures which are not too much larger than man-size. Total the values for each group & ratio one over the other. [Seriously? ed.] If side A has 4 man-sized participants, & side B has 3 smaller than man-sized participants & 1 size "L" bugbear, the ratio is 4:3. Then, according to the direction of the missile discharge, determine hits by using the same ratio. If 7 missiles were loosed, 4 would have a chance to hit side A, 3 side B... [there's a bit more of the same, then] You may assign a minor chance of the missile striking a friend if you wish, but this writer [Gygax], for instance, always allows archery hits to hit a giant or a similar creature engaged against a human or smaller opponent."
MOLDVAY BASIC
------------------------
"Missile fire is a type of combat used when the enemy is too far away to fight hand-to-hand (more than 5 feet away)... is fought @ a range beyond 5 feet but still within sight... The DM must check for surprise (or initiative) to see which side fires first... If the distance between enemies is reduced to 5' or less as a result of movement, hand-to-hand combat may take place."
MENTZER BASIC
------------------------
"Devices cannot be used in hand-to-hand combat, or against targets within 5 feet."
Interestingly, there seems to be an actual evolution of thought here. First, it's no missiles where there's melee then no missiles once melee begins b/c then there seems to be a bit of a sea change on the topic when Gygax works up AD&D strangely stating "it is permissible, of course..." & then goes on to spell out how to do it which proves to be a royal pain in the ass to execute in any decently sized battle... He then refers to the older rules about friendly fire, but tries to simplify the thought by saying, basically, make it easy on yourself, if the monsters you are firing at are much larger than the bowman, dont sweat it... THEN, w/ the following editions of Basic, unless I missed something, you dont see "you can't fire into melee" anymore... Now, it appears permissible, but you must be at least 5 feet away from the target to do so, & also, you must have the initiative*, not be surprised, etc.
My general ruling on this will always have a lot to do w/ what exactly a player tells me they do, how the initiative falls, & if there is any question as to whether there can be success of actually getting an arrow or missile fire off... If so, a Dexterity check may come into play... HOWEVER, if a player gives me solid info to go on beforehand, e.g. I move a bit away from the party, toward the monsters' rear & jump on a table, etc. Generally there will be no questions asked. But when assumed melee combat begins, initiative starts being rolled & a player just says, "I fire my bow" when it gets to them, there are just too many variables that cannot be determined, therefore, it cannot even be an option, especially when the player does not have initiative*. It is mainly just a role-playing thing. If a player has thought out their attack then I should hear about it, but if they havent, it becomes obvious, & therefore their character shouldnt be able to suddenly do so.
So anyway, what if a player is able to fire their missile into melee combat per above? What about "friendly fire"? As the threat should be real, any miss where the to-hit roll (w/ modifiers) is lower than the discharger's DEX score subtracted from 20, has a 10% (1 on 1d10) chance of hitting a friendly PC or NPC; generally the closest to the missed target. Here's an example: Legolas in an impressive feat of dexterity, skirts the melee & lofts himself into the air from a treetrunk, firing off an arrow @ an Orc that is surrounded by 2 friendly fighters. Legolas rolls a 5 to-hit (rolls a 3, but gets +2 for DEX bonus; NOTE: in this instance, missile fire does not get a to-hit bonus for close range - only so when discharger is stationary when firing). Legolas has missed the Orc, therefore you take his DEX score of 16 & subtract it from 20. That tells you that if Legolas' to-hit was below a 4, then there is a chance he has hit one of the friendly fighters. As his to-hit is a 5, he hits no one & battle resumes to the next w/ initiative. That said, no matter what one's DEX or modifiers, if the discharger rolls a natural "1" to-hit, they will always hit a friendly unit (no 1d10 roll necessary), though in some situations a GM may devise another situation they find more appropriate.
*as for Initiative, I do it a bit differently than most I'm sure, perhaps it's my experience w/ other games, but the closest D&D "option" would be Moldvay's "Pair Combat" rule. In that variant, instead of each "side" rolling for initiative & one "side" attacking after the other, you pair off one member from each party attacking another member of the opposite party & then each "pair" rolls for initiative against one another. The way I do it, however, is w/o the pairing. Instead, every character & monster rolls for initiative, adds any modifiers, & then the combat executes from 6 down to 1. Personally, I feel this gives a more visual flow to combat. Someone may get a jump that usually doesnt, etc. (However, two-handed weapon wielders always go off @ 1, unless they have DEX modifier - that is, if using variable weapon damage)
