Getting around the argument list too long problem in linux
copied from LinuxJournal.com
by A. Naro
Method #1: Manually split the command line arguments into smaller bunches.
Example 1
[user@localhost directory]$ mv [a-l]* ../directory2
[user@localhost directory]$ mv [m-z]* ../directory2
Method #2: Use the find command.
Example 2
[user@localhost directory]$ find $directory -type f -name '*' -exec mv
{} $directory2/. ;
Method #3: Create a function. *
Example 3a
function large_mv ()
{ while read line1; do
mv directory/$line1 ../directory2
done
}
ls -1 directory/ | large_mv
Example 3b
function larger_mv ()
{ while read line1; do
md5sum directory/$line1 >> ~/md5sums
ls -l directory/$line1 >> ~/backup_list
mv directory/$line1 ../directory2
done
}
ls -1 directory/ | larger_mv
Example 3b demonstrates how you easily can get an md5sum and a backup listing of each file before moving it.
Method #4: Recompile the Linux kernel. **This last method requires a word of caution, as it is by far the most aggressive solution to the problem. It is presented here for the sake of thoroughness, since it is a valid method of getting around the problem. However, please be advised that due to the advanced nature of the solution, only experienced Linux users should attempt this hack. In addition, make sure to thoroughly test the final result in your environment before implementing it permanently.
One of the advantages of using an open-source kernel is that you are able to examine exactly what it is configured to do and modify its parameters to suit the individual needs of your system. Method #4 involves manually increasing the number of pages that are allocated within the kernel for command-line arguments. If you look at the include/linux/binfmts.h file, you will find the following near the top:
/* * MAX_ARG_PAGES defines the number of pages allocated for arguments * and envelope for the new program. 32 should suffice, this gives * a maximum env+arg of 128kB w/4KB pages! */ #define MAX_ARG_PAGES 32
In order to increase the amount of memory dedicated to the command-line arguments, you simply need to provide the MAX_ARG_PAGES value with a higher number. Once this edit is saved, simply recompile, install and reboot into the new kernel as you would do normally.
On my own test system I managed to solve all my problems by raising this value to 64. After extensive testing, I have not experienced a single problem since the switch. This is entirely expected since even with MAX_ARG_PAGES set to 64, the longest possible command line I could produce would only occupy 256KB of system memory–not very much by today’s system hardware standards.
The advantages of Method #4 are clear. You are now able to simply run the command as you would normally, and it completes successfully. The disadvantages are equally clear. If you raise the amount of memory available to the command line beyond the amount of available system memory, you can create a D.O.S. attack on your own system and cause it to crash. On multiuser systems in particular, even a small increase can have a significant impact because every user is then allocated the additional memory. Therefore always test extensively in your own environment, as this is the safest way to determine if Method #4 is a viable option for you.