Unable to VirtualAllocEx more than 32664 items.

B

Bob Karaban

We ran into a problem using VirtualAllocEx and were wondering if anybody has
a way around this. We have an executable that stores a hash table in a
remote process. The VirtualAllocEx function fails on the 32665 item. Below
is a sample project to show it.


Program 1 (where the memory will be stored)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>
#include <iostream.h>


int main(int argc, char* argv[])
{
cout << "Process ID: " << GetCurrentProcessId() << endl;
HANDLE object = CreateSemaphore(NULL,0,1,NULL);
WaitForSingleObject(object,INFINITE);
return 0;
}


Program 2 (who will do the allocating - Note: update PID to the pid of the
above code)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>


// Main program

int main(int argc, char* argv[])
{
int i=0;
long err = 0;
HANDLE hProcess = 0;
long remoteMemory[100000];
char* srcMemory = new char[1024];
memset(&srcMemory[0],75,1024);


////////////
// Connect // Make sure you change the pid
hProcess =
OpenProcess(PROCESS_VM_OPERATION|PROCESS_VM_READ|PROCESS_VM_WRITE,
FALSE, <CHANGE TO YOUR PID>);
////////////
// Allocate
for (i=0; i<100000; i++)
{
remoteMemory =
(long)VirtualAllocEx(hProcess,NULL,1024,MEM_COMMIT,PAGE_READWRITE);
if (remoteMemory == 0)
err = GetLastError(); // Create
break point here
// you should get err = 8 -->
ERROR_NOT_ENOUGH_MEMORY
}

////////////
// Free
for (i=0;i<100000;i++)

VirtualFreeEx(hProcess,(void*)remoteMemory,0,MEM_RELEASE);

CloseHandle(hProcess);
return 0;
}

Does anyone know where this limit of 32664 is coming from (almost an int)?
Is there some way around this limitation? Any help would be greatly
appreciated. Feel free to contact me back at any of these newsgroups or
directly via email - thanks in advance!



Bob
 
R

red floyd

Bob said:
We ran into a problem using VirtualAllocEx and were wondering if anybody has
a way around this. We have an executable that stores a hash table in a
remote process. The VirtualAllocEx function fails on the 32665 item. Below
is a sample project to show it.


Program 1 (where the memory will be stored)
Click to expand...

My compiler has no clue what VirtualAllocEx is. You might try asking in
a VC newsgroup.
 
D

Dave Townsend

Just a guess, but VirtualAllocEx allocates in complete pages, so even though
you ask for 1k, you are getting a whole page worth, 8k ? That adds up to
256Meg,
do you have that much memory on your system available ?

dave
 
B

Bob Karaban

Actually, we do have the memory available - originally, we were thinking the
same thing. What we tried was changing how much memory we allocated - it
doesn't matter if we allocate in 1k blocks or 100k blocks, we can only do it
32,664 times! So, the amount of total memory we use may vary wildly (we
never actually run out) depending on the block size, but we never complete
request #32,665! Very strange! Any other guesses, please keep them coming!

Bob



Dave Townsend said:
Just a guess, but VirtualAllocEx allocates in complete pages, so even though
you ask for 1k, you are getting a whole page worth, 8k ? That adds up to
256Meg,
do you have that much memory on your system available ?

dave


Bob Karaban said:
We ran into a problem using VirtualAllocEx and were wondering if anybody has
a way around this. We have an executable that stores a hash table in a
remote process. The VirtualAllocEx function fails on the 32665 item. Below
is a sample project to show it.


Program 1 (where the memory will be stored)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>
#include <iostream.h>


int main(int argc, char* argv[])
{
cout << "Process ID: " << GetCurrentProcessId() << endl;
HANDLE object = CreateSemaphore(NULL,0,1,NULL);
WaitForSingleObject(object,INFINITE);
return 0;
}


Program 2 (who will do the allocating - Note: update PID to the pid of the
above code)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>


// Main program

int main(int argc, char* argv[])
{
int i=0;
long err = 0;
HANDLE hProcess = 0;
long remoteMemory[100000];
char* srcMemory = new char[1024];
memset(&srcMemory[0],75,1024);


////////////
// Connect // Make sure you change the pid
hProcess =
OpenProcess(PROCESS_VM_OPERATION|PROCESS_VM_READ|PROCESS_VM_WRITE,
FALSE, <CHANGE TO YOUR PID>);
////////////
// Allocate
for (i=0; i<100000; i++)
{
remoteMemory =
(long)VirtualAllocEx(hProcess,NULL,1024,MEM_COMMIT,PAGE_READWRITE);
if (remoteMemory == 0)
err = GetLastError(); // Create
break point here
// you should get err = 8 -->
ERROR_NOT_ENOUGH_MEMORY
}

////////////
// Free
for (i=0;i<100000;i++)

VirtualFreeEx(hProcess,(void*)remoteMemory,0,MEM_RELEASE);

CloseHandle(hProcess);
return 0;
}

Does anyone know where this limit of 32664 is coming from (almost an int)?
Is there some way around this limitation? Any help would be greatly
appreciated. Feel free to contact me back at any of these newsgroups or
directly via email - thanks in advance!



Bob
Click to expand...
Click to expand...
 
J

Jochen Kalmbach

Bob said:
We ran into a problem using VirtualAllocEx and were wondering if
anybody has a way around this. We have an executable that stores a
hash table in a remote process. The VirtualAllocEx function fails on
the 32665 item. Below is a sample project to show it.
Click to expand...

Just a guess: Maybe your LDT is full !?

--
Greetings
Jochen

Do you need a memory-leak finder ?
http://www.codeproject.com/tools/leakfinder.asp

Do you need daily reports from your server?
http://sourceforge.net/projects/srvreport/
 
I

Ivan Brugiolo [MSFT]

VirtualAlloc, as per the documentation, allocates with a certain
granularity, that, in your case should be 64K.

0:000> ?0n32665*0x10000
Evaluate expression: 2140733440 = 7f990000 // this is about 2 Gigs
0:000>

in this case, you have simply exausted the whole virtual address space of
your process.

On average, you should use the !address command in cdb/ntsd/windbg to
diagnose these problems more easily.
 
B

Bob Karaban

O.K., we think we understand - how would we reduce this granularity? If we
reduce the granularity, we should then be able to get proportionally more
VirtualAllocEx calls, correct?

Thanks for the help!


Bob



Ivan Brugiolo said:
VirtualAlloc, as per the documentation, allocates with a certain
granularity, that, in your case should be 64K.

0:000> ?0n32665*0x10000
Evaluate expression: 2140733440 = 7f990000 // this is about 2 Gigs
0:000>

in this case, you have simply exausted the whole virtual address space of
your process.

On average, you should use the !address command in cdb/ntsd/windbg to
diagnose these problems more easily.

--
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of any included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm


Bob Karaban said:
We ran into a problem using VirtualAllocEx and were wondering if anybody has
a way around this. We have an executable that stores a hash table in a
remote process. The VirtualAllocEx function fails on the 32665 item. Below
is a sample project to show it.


Program 1 (where the memory will be stored)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>
#include <iostream.h>


int main(int argc, char* argv[])
{
cout << "Process ID: " << GetCurrentProcessId() << endl;
HANDLE object = CreateSemaphore(NULL,0,1,NULL);
WaitForSingleObject(object,INFINITE);
return 0;
}


Program 2 (who will do the allocating - Note: update PID to the pid of the
above code)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>


// Main program

int main(int argc, char* argv[])
{
int i=0;
long err = 0;
HANDLE hProcess = 0;
long remoteMemory[100000];
char* srcMemory = new char[1024];
memset(&srcMemory[0],75,1024);


////////////
// Connect // Make sure you change the pid
hProcess =
OpenProcess(PROCESS_VM_OPERATION|PROCESS_VM_READ|PROCESS_VM_WRITE,
FALSE, <CHANGE TO YOUR PID>);
////////////
// Allocate
for (i=0; i<100000; i++)
{
remoteMemory =
(long)VirtualAllocEx(hProcess,NULL,1024,MEM_COMMIT,PAGE_READWRITE);
if (remoteMemory == 0)
err = GetLastError(); // Create
break point here
// you should get err = 8 -->
ERROR_NOT_ENOUGH_MEMORY
}

////////////
// Free
for (i=0;i<100000;i++)

VirtualFreeEx(hProcess,(void*)remoteMemory,0,MEM_RELEASE);

CloseHandle(hProcess);
return 0;
}

Does anyone know where this limit of 32664 is coming from (almost an int)?
Is there some way around this limitation? Any help would be greatly
appreciated. Feel free to contact me back at any of these newsgroups or
directly via email - thanks in advance!



Bob
Click to expand...
Click to expand...
 
D

Dave Townsend

Yeah, I this makes sense, I just looked at the .NET documentation on
virtualallocex, it says the desired
address is rounded to the nearest 64k boundary, so it sounds like you've
blown
the addressability range of windows32.

Since you only need 1k for each entry, can you do some clever stuff to split
up
the 64k page amongst 64 separate entries perhaps ?

dave


Ivan Brugiolo said:
VirtualAlloc, as per the documentation, allocates with a certain
granularity, that, in your case should be 64K.

0:000> ?0n32665*0x10000
Evaluate expression: 2140733440 = 7f990000 // this is about 2 Gigs
0:000>

in this case, you have simply exausted the whole virtual address space of
your process.

On average, you should use the !address command in cdb/ntsd/windbg to
diagnose these problems more easily.

--
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of any included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm


Bob Karaban said:
We ran into a problem using VirtualAllocEx and were wondering if anybody has
a way around this. We have an executable that stores a hash table in a
remote process. The VirtualAllocEx function fails on the 32665 item. Below
is a sample project to show it.


Program 1 (where the memory will be stored)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>
#include <iostream.h>


int main(int argc, char* argv[])
{
cout << "Process ID: " << GetCurrentProcessId() << endl;
HANDLE object = CreateSemaphore(NULL,0,1,NULL);
WaitForSingleObject(object,INFINITE);
return 0;
}


Program 2 (who will do the allocating - Note: update PID to the pid of the
above code)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>


// Main program

int main(int argc, char* argv[])
{
int i=0;
long err = 0;
HANDLE hProcess = 0;
long remoteMemory[100000];
char* srcMemory = new char[1024];
memset(&srcMemory[0],75,1024);


////////////
// Connect // Make sure you change the pid
hProcess =
OpenProcess(PROCESS_VM_OPERATION|PROCESS_VM_READ|PROCESS_VM_WRITE,
FALSE, <CHANGE TO YOUR PID>);
////////////
// Allocate
for (i=0; i<100000; i++)
{
remoteMemory =
(long)VirtualAllocEx(hProcess,NULL,1024,MEM_COMMIT,PAGE_READWRITE);
if (remoteMemory == 0)
err = GetLastError(); // Create
break point here
// you should get err = 8 -->
ERROR_NOT_ENOUGH_MEMORY
}

////////////
// Free
for (i=0;i<100000;i++)

VirtualFreeEx(hProcess,(void*)remoteMemory,0,MEM_RELEASE);

CloseHandle(hProcess);
return 0;
}

Does anyone know where this limit of 32664 is coming from (almost an int)?
Is there some way around this limitation? Any help would be greatly
appreciated. Feel free to contact me back at any of these newsgroups or
directly via email - thanks in advance!



Bob
Click to expand...
Click to expand...
 
I

Ivan Brugiolo [MSFT]

The granularity is pretty much hard-coded in the memory manager
per each OS and platform and architecture.
You should consider virtualallocating a 64K chunks in the reserved state,
and then commit the individual pages on demand.

--
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of any included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm


Bob Karaban said:
O.K., we think we understand - how would we reduce this granularity? If we
reduce the granularity, we should then be able to get proportionally more
VirtualAllocEx calls, correct?

Thanks for the help!


Bob



Ivan Brugiolo said:
VirtualAlloc, as per the documentation, allocates with a certain
granularity, that, in your case should be 64K.

0:000> ?0n32665*0x10000
Evaluate expression: 2140733440 = 7f990000 // this is about 2 Gigs
0:000>

in this case, you have simply exausted the whole virtual address space of
your process.

On average, you should use the !address command in cdb/ntsd/windbg to
diagnose these problems more easily.

--
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of any included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm


We ran into a problem using VirtualAllocEx and were wondering if
Click to expand...
anybody
has
a way around this. We have an executable that stores a hash table in a
remote process. The VirtualAllocEx function fails on the 32665 item. Below
is a sample project to show it.


Program 1 (where the memory will be stored)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>
#include <iostream.h>


int main(int argc, char* argv[])
{
cout << "Process ID: " << GetCurrentProcessId() << endl;
HANDLE object = CreateSemaphore(NULL,0,1,NULL);
WaitForSingleObject(object,INFINITE);
return 0;
}


Program 2 (who will do the allocating - Note: update PID to the pid of the
above code)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>


// Main program

int main(int argc, char* argv[])
{
int i=0;
long err = 0;
HANDLE hProcess = 0;
long remoteMemory[100000];
char* srcMemory = new char[1024];
memset(&srcMemory[0],75,1024);


////////////
// Connect // Make sure you change the pid
hProcess =
OpenProcess(PROCESS_VM_OPERATION|PROCESS_VM_READ|PROCESS_VM_WRITE,
FALSE, <CHANGE TO YOUR PID>);
////////////
// Allocate
for (i=0; i<100000; i++)
{
remoteMemory =
(long)VirtualAllocEx(hProcess,NULL,1024,MEM_COMMIT,PAGE_READWRITE);
if (remoteMemory == 0)
err = GetLastError(); // Create
break point here
// you should get err = 8 -->
ERROR_NOT_ENOUGH_MEMORY
}

////////////
// Free
for (i=0;i<100000;i++)

VirtualFreeEx(hProcess,(void*)remoteMemory,0,MEM_RELEASE);

CloseHandle(hProcess);
return 0;
}

Does anyone know where this limit of 32664 is coming from (almost an int)?
Is there some way around this limitation? Any help would be greatly
appreciated. Feel free to contact me back at any of these newsgroups or
directly via email - thanks in advance!



Bob
Click to expand...
Click to expand...
Click to expand...
 
S

Severian

We ran into a problem using VirtualAllocEx and were wondering if anybody has
a way around this. We have an executable that stores a hash table in a
remote process. The VirtualAllocEx function fails on the 32665 item. Below
is a sample project to show it.


Program 1 (where the memory will be stored)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>
#include <iostream.h>


int main(int argc, char* argv[])
{
cout << "Process ID: " << GetCurrentProcessId() << endl;
HANDLE object = CreateSemaphore(NULL,0,1,NULL);
WaitForSingleObject(object,INFINITE);
return 0;
}


Program 2 (who will do the allocating - Note: update PID to the pid of the
above code)
---------------------------------------------
#include "stdafx.h"
#include <windows.h>


// Main program

int main(int argc, char* argv[])
{
int i=0;
long err = 0;
HANDLE hProcess = 0;
long remoteMemory[100000];
char* srcMemory = new char[1024];
memset(&srcMemory[0],75,1024);


////////////
// Connect // Make sure you change the pid
hProcess =
OpenProcess(PROCESS_VM_OPERATION|PROCESS_VM_READ|PROCESS_VM_WRITE,
FALSE, <CHANGE TO YOUR PID>);
////////////
// Allocate
for (i=0; i<100000; i++)
{
remoteMemory =
(long)VirtualAllocEx(hProcess,NULL,1024,MEM_COMMIT,PAGE_READWRITE);
if (remoteMemory == 0)
err = GetLastError(); // Create
break point here
// you should get err = 8 -->
ERROR_NOT_ENOUGH_MEMORY
}

////////////
// Free
for (i=0;i<100000;i++)

VirtualFreeEx(hProcess,(void*)remoteMemory,0,MEM_RELEASE);

CloseHandle(hProcess);
return 0;
}

Does anyone know where this limit of 32664 is coming from (almost an int)?
Is there some way around this limitation? Any help would be greatly
appreciated. Feel free to contact me back at any of these newsgroups or
directly via email - thanks in advance!
Click to expand...


If you're writing using C, use malloc() & free. If you insist on Win32
API, try HeapAlloc & friends.

IIRC, both only call VirtualAlloc(Ex) when necessary and subdivide the
large blocks.
 
S

Severian

Why are you crossposting this drivel to comp.lang.c++?
Click to expand...

I at least apologize abjectly. With only a cursory glance prior to my
earlier response, I thought they were all MS shite. I'm sorry for my
contribution to the pollution in comp.lang.c++!
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads

Windows LLDP Driver Responds With No Data 0
Communicating between processes 0
Linux: using "clone3" and "waitid" 0
More than one instance. 18
Adding adressing of IPv6 to program 1
I try to make a stb stile one header file library and can't get it to work 1
Please help with C programming to save GPS reception data in Raspberry Pi. 0
How can I view / open / render / display a pdf file with c code? 0

Members online

Total: 70 (members: 2, guests: 68)
Robots: 455

Forum statistics

Threads
473,995
Messages
2,570,233
Members
46,820
Latest member
GilbertoA5

Latest Threads

Top