It's true that about 7 million working-age men have dropped out of the labor force. But the reason isn't competition with immigrants for jobs. It's mostly illness and disability.

Where Are The 7 Million Missing Men?
You may have heard that there are roughly 7 million men of working age that are not currently in the labor force -- that is, not currently working or looking for work. The statistic has been produced in various ways using slightly different definitions by different researchers, but the most well-known is from Nicholas Eberstadt…
economistwritingeveryday.com