Linux: Cannot allocate more than 32 GB/64 GB of memory in a single process due to virtual memory limit -

i have computer 128 gb of ram, running linux (3.19.5-200.fc21.x86_64). however, cannot allocate more ~30 gb of ram in single process. beyond this, malloc fails:

#include <stdlib.h> #include <iostream>  int main() {    size_t gb_in_bytes = size_t(1)<<size_t(30); // 1 gb in bytes (2^30).    // try allocate 1 block of 'i' gb.    (size_t = 25; < 35; ++ i) {       size_t n = * gb_in_bytes;       void *p = ::malloc(n);       std::cout << "allocation of 1 x " << (n/double(gb_in_bytes)) << " gb of data. ok? " << ((p==0)? "nope" : "yes") << std::endl;       ::free(p);    } } 

this produces following output:

/tmp> c++ mem_alloc.cpp && a.out  allocation of 1 x 25 gb of data. ok? yes allocation of 1 x 26 gb of data. ok? yes allocation of 1 x 27 gb of data. ok? yes allocation of 1 x 28 gb of data. ok? yes allocation of 1 x 29 gb of data. ok? yes allocation of 1 x 30 gb of data. ok? yes allocation of 1 x 31 gb of data. ok? nope allocation of 1 x 32 gb of data. ok? nope allocation of 1 x 33 gb of data. ok? nope allocation of 1 x 34 gb of data. ok? nope 

i searched quite time, , found related maximum virtual memory size:

~> ulimit -all [...] max locked memory       (kbytes, -l) 64 max memory size         (kbytes, -m) unlimited virtual memory          (kbytes, -v) 32505856 [...] 

i can increase limit ~64 gb via ulimit -v 64000000, not further. beyond this, operation not permitted errors:

~> ulimit -v 64000000 ~> ulimit -v 65000000                                                                                                                                   bash: ulimit: virtual memory: cannot modify limit: operation not permitted                                                                               ~> ulimit -v unlimited bash: ulimit: virtual memory: cannot modify limit: operation not permitted  

some more searching revealed in principle should possible set these limits via "as" (address space) entry in /etc/security/limits.conf. however, doing this, reduce maximum amount of virtual memory, not increase it.

is there way either lift limit of virtual memory per process completely, or increase beyond 64 gb? use of physical memory in single application.

edit:

• following ingo leonhardt, tried ulimits -v unlimited after logging in root, not standard user. doing solves problem root (the program can allocate physical memory while logged in root). works root, not other users. however, @ least means in principle kernel can handle fine, , there configuration problem.

• regarding limits.conf: tried explicitly adding

  • hard unlimited
  • soft unlimited

  to /etc/security/limits.conf, , rebooting. had no effect. after login standard user, ulimit -v still returns 32 gb, , ulimit -v 65000000 still says permission denied (while ulimit -v 64000000 works). rest of limits.conf commented out, , in /etc/security/limits.d there 1 other, unrelated entry (limiting nproc 4096 non-root users). is, virtual memory limit must coming elsewhere limits.conf. ideas else lead ulimits -v not being "unlimited"?

edit/resolution:

• it caused own stupidity. had (long forgotten) program in user setup used setrlimit restrict amount of memory per process prevent linux swapping death. unintentionally copied 32 gb machine 128 gb machine. paul , andrew janke , else helping track down. sorry :/.

• if else encounters this: search ulimit/setrlimit in bash , profile settings, , programs potentially calling (both own , system-wide /etc settings) , make sure /security/limits.conf not include limit... (or @ least try creating new user, see if happens in user or system setup)

this ulimit , system setup problem, not c++ problem.

i can run appropriately modified code on amazon ec2 instance type r3.4xlarge no problem. these cost less $0.20/hour on spot market, , suggest rent one, , perhaps take around in /etc , compare own setup... or maybe need recompile linux kernel use memory... not c++ or gcc problem.

ubuntu on ec2 machine set unlimited process memory.

$ sudo su # ulimit -u --> unlimited 

this 1 has 125gb of ram

# free              total       used       free     shared    buffers     cached mem:     125903992    1371828  124532164        344      22156     502248 -/+ buffers/cache:     847424  125056568 swap:            0          0          0 

i modified limits on program go 149gb.

here's output. looks 118gb.

root@ip-10-203-193-204:/home/ubuntu# ./memtest allocation of 1 x 25 gb of data. ok? yes allocation of 1 x 26 gb of data. ok? yes allocation of 1 x 27 gb of data. ok? yes allocation of 1 x 28 gb of data. ok? yes allocation of 1 x 29 gb of data. ok? yes allocation of 1 x 30 gb of data. ok? yes allocation of 1 x 31 gb of data. ok? yes allocation of 1 x 32 gb of data. ok? yes allocation of 1 x 33 gb of data. ok? yes allocation of 1 x 34 gb of data. ok? yes allocation of 1 x 35 gb of data. ok? yes allocation of 1 x 36 gb of data. ok? yes allocation of 1 x 37 gb of data. ok? yes allocation of 1 x 38 gb of data. ok? yes allocation of 1 x 39 gb of data. ok? yes allocation of 1 x 40 gb of data. ok? yes allocation of 1 x 41 gb of data. ok? yes allocation of 1 x 42 gb of data. ok? yes allocation of 1 x 43 gb of data. ok? yes allocation of 1 x 44 gb of data. ok? yes allocation of 1 x 45 gb of data. ok? yes allocation of 1 x 46 gb of data. ok? yes allocation of 1 x 47 gb of data. ok? yes allocation of 1 x 48 gb of data. ok? yes allocation of 1 x 49 gb of data. ok? yes allocation of 1 x 50 gb of data. ok? yes allocation of 1 x 51 gb of data. ok? yes allocation of 1 x 52 gb of data. ok? yes allocation of 1 x 53 gb of data. ok? yes allocation of 1 x 54 gb of data. ok? yes allocation of 1 x 55 gb of data. ok? yes allocation of 1 x 56 gb of data. ok? yes allocation of 1 x 57 gb of data. ok? yes allocation of 1 x 58 gb of data. ok? yes allocation of 1 x 59 gb of data. ok? yes allocation of 1 x 60 gb of data. ok? yes allocation of 1 x 61 gb of data. ok? yes allocation of 1 x 62 gb of data. ok? yes allocation of 1 x 63 gb of data. ok? yes allocation of 1 x 64 gb of data. ok? yes allocation of 1 x 65 gb of data. ok? yes allocation of 1 x 66 gb of data. ok? yes allocation of 1 x 67 gb of data. ok? yes allocation of 1 x 68 gb of data. ok? yes allocation of 1 x 69 gb of data. ok? yes allocation of 1 x 70 gb of data. ok? yes allocation of 1 x 71 gb of data. ok? yes allocation of 1 x 72 gb of data. ok? yes allocation of 1 x 73 gb of data. ok? yes allocation of 1 x 74 gb of data. ok? yes allocation of 1 x 75 gb of data. ok? yes allocation of 1 x 76 gb of data. ok? yes allocation of 1 x 77 gb of data. ok? yes allocation of 1 x 78 gb of data. ok? yes allocation of 1 x 79 gb of data. ok? yes allocation of 1 x 80 gb of data. ok? yes allocation of 1 x 81 gb of data. ok? yes allocation of 1 x 82 gb of data. ok? yes allocation of 1 x 83 gb of data. ok? yes allocation of 1 x 84 gb of data. ok? yes allocation of 1 x 85 gb of data. ok? yes allocation of 1 x 86 gb of data. ok? yes allocation of 1 x 87 gb of data. ok? yes allocation of 1 x 88 gb of data. ok? yes allocation of 1 x 89 gb of data. ok? yes allocation of 1 x 90 gb of data. ok? yes allocation of 1 x 91 gb of data. ok? yes allocation of 1 x 92 gb of data. ok? yes allocation of 1 x 93 gb of data. ok? yes allocation of 1 x 94 gb of data. ok? yes allocation of 1 x 95 gb of data. ok? yes allocation of 1 x 96 gb of data. ok? yes allocation of 1 x 97 gb of data. ok? yes allocation of 1 x 98 gb of data. ok? yes allocation of 1 x 99 gb of data. ok? yes allocation of 1 x 100 gb of data. ok? yes allocation of 1 x 101 gb of data. ok? yes allocation of 1 x 102 gb of data. ok? yes allocation of 1 x 103 gb of data. ok? yes allocation of 1 x 104 gb of data. ok? yes allocation of 1 x 105 gb of data. ok? yes allocation of 1 x 106 gb of data. ok? yes allocation of 1 x 107 gb of data. ok? yes allocation of 1 x 108 gb of data. ok? yes allocation of 1 x 109 gb of data. ok? yes allocation of 1 x 110 gb of data. ok? yes allocation of 1 x 111 gb of data. ok? yes allocation of 1 x 112 gb of data. ok? yes allocation of 1 x 113 gb of data. ok? yes allocation of 1 x 114 gb of data. ok? yes allocation of 1 x 115 gb of data. ok? yes allocation of 1 x 116 gb of data. ok? yes allocation of 1 x 117 gb of data. ok? yes allocation of 1 x 118 gb of data. ok? yes allocation of 1 x 119 gb of data. ok? nope allocation of 1 x 120 gb of data. ok? nope allocation of 1 x 121 gb of data. ok? nope allocation of 1 x 122 gb of data. ok? nope allocation of 1 x 123 gb of data. ok? nope allocation of 1 x 124 gb of data. ok? nope allocation of 1 x 125 gb of data. ok? nope allocation of 1 x 126 gb of data. ok? nope allocation of 1 x 127 gb of data. ok? nope allocation of 1 x 128 gb of data. ok? nope allocation of 1 x 129 gb of data. ok? nope allocation of 1 x 130 gb of data. ok? nope allocation of 1 x 131 gb of data. ok? nope allocation of 1 x 132 gb of data. ok? nope allocation of 1 x 133 gb of data. ok? nope allocation of 1 x 134 gb of data. ok? nope allocation of 1 x 135 gb of data. ok? nope allocation of 1 x 136 gb of data. ok? nope allocation of 1 x 137 gb of data. ok? nope allocation of 1 x 138 gb of data. ok? nope allocation of 1 x 139 gb of data. ok? nope allocation of 1 x 140 gb of data. ok? nope allocation of 1 x 141 gb of data. ok? nope allocation of 1 x 142 gb of data. ok? nope allocation of 1 x 143 gb of data. ok? nope allocation of 1 x 144 gb of data. ok? nope allocation of 1 x 145 gb of data. ok? nope allocation of 1 x 146 gb of data. ok? nope allocation of 1 x 147 gb of data. ok? nope allocation of 1 x 148 gb of data. ok? nope allocation of 1 x 149 gb of data. ok? nope 

now, us$0.17 spent on this...