Other games have systems that kick players from activities for long periods of inactivity. Those who are kicked are soon replaced by others who have started looking for matches. Maybe something like that would help.