VirtualBox

source: vbox/trunk/src/VBox/VMM/VMMR0/GMMR0.cpp@ 62478

Last change on this file since 62478 was 62478, checked in by vboxsync, 8 years ago

(C) 2016

  • Property svn:eol-style set to native
  • Property svn:keywords set to Id Revision
File size: 190.9 KB
Line 
1/* $Id: GMMR0.cpp 62478 2016-07-22 18:29:06Z vboxsync $ */
2/** @file
3 * GMM - Global Memory Manager.
4 */
5
6/*
7 * Copyright (C) 2007-2016 Oracle Corporation
8 *
9 * This file is part of VirtualBox Open Source Edition (OSE), as
10 * available from http://www.virtualbox.org. This file is free software;
11 * you can redistribute it and/or modify it under the terms of the GNU
12 * General Public License (GPL) as published by the Free Software
13 * Foundation, in version 2 as it comes in the "COPYING" file of the
14 * VirtualBox OSE distribution. VirtualBox OSE is distributed in the
15 * hope that it will be useful, but WITHOUT ANY WARRANTY of any kind.
16 */
17
18
19/** @page pg_gmm GMM - The Global Memory Manager
20 *
21 * As the name indicates, this component is responsible for global memory
22 * management. Currently only guest RAM is allocated from the GMM, but this
23 * may change to include shadow page tables and other bits later.
24 *
25 * Guest RAM is managed as individual pages, but allocated from the host OS
26 * in chunks for reasons of portability / efficiency. To minimize the memory
27 * footprint all tracking structure must be as small as possible without
28 * unnecessary performance penalties.
29 *
30 * The allocation chunks has fixed sized, the size defined at compile time
31 * by the #GMM_CHUNK_SIZE \#define.
32 *
33 * Each chunk is given an unique ID. Each page also has a unique ID. The
34 * relation ship between the two IDs is:
35 * @code
36 * GMM_CHUNK_SHIFT = log2(GMM_CHUNK_SIZE / PAGE_SIZE);
37 * idPage = (idChunk << GMM_CHUNK_SHIFT) | iPage;
38 * @endcode
39 * Where iPage is the index of the page within the chunk. This ID scheme
40 * permits for efficient chunk and page lookup, but it relies on the chunk size
41 * to be set at compile time. The chunks are organized in an AVL tree with their
42 * IDs being the keys.
43 *
44 * The physical address of each page in an allocation chunk is maintained by
45 * the #RTR0MEMOBJ and obtained using #RTR0MemObjGetPagePhysAddr. There is no
46 * need to duplicate this information (it'll cost 8-bytes per page if we did).
47 *
48 * So what do we need to track per page? Most importantly we need to know
49 * which state the page is in:
50 * - Private - Allocated for (eventually) backing one particular VM page.
51 * - Shared - Readonly page that is used by one or more VMs and treated
52 * as COW by PGM.
53 * - Free - Not used by anyone.
54 *
55 * For the page replacement operations (sharing, defragmenting and freeing)
56 * to be somewhat efficient, private pages needs to be associated with a
57 * particular page in a particular VM.
58 *
59 * Tracking the usage of shared pages is impractical and expensive, so we'll
60 * settle for a reference counting system instead.
61 *
62 * Free pages will be chained on LIFOs
63 *
64 * On 64-bit systems we will use a 64-bit bitfield per page, while on 32-bit
65 * systems a 32-bit bitfield will have to suffice because of address space
66 * limitations. The #GMMPAGE structure shows the details.
67 *
68 *
69 * @section sec_gmm_alloc_strat Page Allocation Strategy
70 *
71 * The strategy for allocating pages has to take fragmentation and shared
72 * pages into account, or we may end up with with 2000 chunks with only
73 * a few pages in each. Shared pages cannot easily be reallocated because
74 * of the inaccurate usage accounting (see above). Private pages can be
75 * reallocated by a defragmentation thread in the same manner that sharing
76 * is done.
77 *
78 * The first approach is to manage the free pages in two sets depending on
79 * whether they are mainly for the allocation of shared or private pages.
80 * In the initial implementation there will be almost no possibility for
81 * mixing shared and private pages in the same chunk (only if we're really
82 * stressed on memory), but when we implement forking of VMs and have to
83 * deal with lots of COW pages it'll start getting kind of interesting.
84 *
85 * The sets are lists of chunks with approximately the same number of
86 * free pages. Say the chunk size is 1MB, meaning 256 pages, and a set
87 * consists of 16 lists. So, the first list will contain the chunks with
88 * 1-7 free pages, the second covers 8-15, and so on. The chunks will be
89 * moved between the lists as pages are freed up or allocated.
90 *
91 *
92 * @section sec_gmm_costs Costs
93 *
94 * The per page cost in kernel space is 32-bit plus whatever RTR0MEMOBJ
95 * entails. In addition there is the chunk cost of approximately
96 * (sizeof(RT0MEMOBJ) + sizeof(CHUNK)) / 2^CHUNK_SHIFT bytes per page.
97 *
98 * On Windows the per page #RTR0MEMOBJ cost is 32-bit on 32-bit windows
99 * and 64-bit on 64-bit windows (a PFN_NUMBER in the MDL). So, 64-bit per page.
100 * The cost on Linux is identical, but here it's because of sizeof(struct page *).
101 *
102 *
103 * @section sec_gmm_legacy Legacy Mode for Non-Tier-1 Platforms
104 *
105 * In legacy mode the page source is locked user pages and not
106 * #RTR0MemObjAllocPhysNC, this means that a page can only be allocated
107 * by the VM that locked it. We will make no attempt at implementing
108 * page sharing on these systems, just do enough to make it all work.
109 *
110 *
111 * @subsection sub_gmm_locking Serializing
112 *
113 * One simple fast mutex will be employed in the initial implementation, not
114 * two as mentioned in @ref sec_pgmPhys_Serializing.
115 *
116 * @see @ref sec_pgmPhys_Serializing
117 *
118 *
119 * @section sec_gmm_overcommit Memory Over-Commitment Management
120 *
121 * The GVM will have to do the system wide memory over-commitment
122 * management. My current ideas are:
123 * - Per VM oc policy that indicates how much to initially commit
124 * to it and what to do in a out-of-memory situation.
125 * - Prevent overtaxing the host.
126 *
127 * There are some challenges here, the main ones are configurability and
128 * security. Should we for instance permit anyone to request 100% memory
129 * commitment? Who should be allowed to do runtime adjustments of the
130 * config. And how to prevent these settings from being lost when the last
131 * VM process exits? The solution is probably to have an optional root
132 * daemon the will keep VMMR0.r0 in memory and enable the security measures.
133 *
134 *
135 *
136 * @section sec_gmm_numa NUMA
137 *
138 * NUMA considerations will be designed and implemented a bit later.
139 *
140 * The preliminary guesses is that we will have to try allocate memory as
141 * close as possible to the CPUs the VM is executed on (EMT and additional CPU
142 * threads). Which means it's mostly about allocation and sharing policies.
143 * Both the scheduler and allocator interface will to supply some NUMA info
144 * and we'll need to have a way to calc access costs.
145 *
146 */
147
148
149/*********************************************************************************************************************************
150* Header Files *
151*********************************************************************************************************************************/
152#define LOG_GROUP LOG_GROUP_GMM
153#include <VBox/rawpci.h>
154#include <VBox/vmm/vm.h>
155#include <VBox/vmm/gmm.h>
156#include "GMMR0Internal.h"
157#include <VBox/vmm/gvm.h>
158#include <VBox/vmm/pgm.h>
159#include <VBox/log.h>
160#include <VBox/param.h>
161#include <VBox/err.h>
162#include <iprt/asm.h>
163#include <iprt/avl.h>
164#ifdef VBOX_STRICT
165# include <iprt/crc.h>
166#endif
167#include <iprt/critsect.h>
168#include <iprt/list.h>
169#include <iprt/mem.h>
170#include <iprt/memobj.h>
171#include <iprt/mp.h>
172#include <iprt/semaphore.h>
173#include <iprt/string.h>
174#include <iprt/time.h>
175
176
177/*********************************************************************************************************************************
178* Defined Constants And Macros *
179*********************************************************************************************************************************/
180/** @def VBOX_USE_CRIT_SECT_FOR_GIANT
181 * Use a critical section instead of a fast mutex for the giant GMM lock.
182 *
183 * @remarks This is primarily a way of avoiding the deadlock checks in the
184 * windows driver verifier. */
185#if defined(RT_OS_WINDOWS) || defined(DOXYGEN_RUNNING)
186# define VBOX_USE_CRIT_SECT_FOR_GIANT
187#endif
188
189
190/*********************************************************************************************************************************
191* Structures and Typedefs *
192*********************************************************************************************************************************/
193/** Pointer to set of free chunks. */
194typedef struct GMMCHUNKFREESET *PGMMCHUNKFREESET;
195
196/**
197 * The per-page tracking structure employed by the GMM.
198 *
199 * On 32-bit hosts we'll some trickery is necessary to compress all
200 * the information into 32-bits. When the fSharedFree member is set,
201 * the 30th bit decides whether it's a free page or not.
202 *
203 * Because of the different layout on 32-bit and 64-bit hosts, macros
204 * are used to get and set some of the data.
205 */
206typedef union GMMPAGE
207{
208#if HC_ARCH_BITS == 64
209 /** Unsigned integer view. */
210 uint64_t u;
211
212 /** The common view. */
213 struct GMMPAGECOMMON
214 {
215 uint32_t uStuff1 : 32;
216 uint32_t uStuff2 : 30;
217 /** The page state. */
218 uint32_t u2State : 2;
219 } Common;
220
221 /** The view of a private page. */
222 struct GMMPAGEPRIVATE
223 {
224 /** The guest page frame number. (Max addressable: 2 ^ 44 - 16) */
225 uint32_t pfn;
226 /** The GVM handle. (64K VMs) */
227 uint32_t hGVM : 16;
228 /** Reserved. */
229 uint32_t u16Reserved : 14;
230 /** The page state. */
231 uint32_t u2State : 2;
232 } Private;
233
234 /** The view of a shared page. */
235 struct GMMPAGESHARED
236 {
237 /** The host page frame number. (Max addressable: 2 ^ 44 - 16) */
238 uint32_t pfn;
239 /** The reference count (64K VMs). */
240 uint32_t cRefs : 16;
241 /** Used for debug checksumming. */
242 uint32_t u14Checksum : 14;
243 /** The page state. */
244 uint32_t u2State : 2;
245 } Shared;
246
247 /** The view of a free page. */
248 struct GMMPAGEFREE
249 {
250 /** The index of the next page in the free list. UINT16_MAX is NIL. */
251 uint16_t iNext;
252 /** Reserved. Checksum or something? */
253 uint16_t u16Reserved0;
254 /** Reserved. Checksum or something? */
255 uint32_t u30Reserved1 : 30;
256 /** The page state. */
257 uint32_t u2State : 2;
258 } Free;
259
260#else /* 32-bit */
261 /** Unsigned integer view. */
262 uint32_t u;
263
264 /** The common view. */
265 struct GMMPAGECOMMON
266 {
267 uint32_t uStuff : 30;
268 /** The page state. */
269 uint32_t u2State : 2;
270 } Common;
271
272 /** The view of a private page. */
273 struct GMMPAGEPRIVATE
274 {
275 /** The guest page frame number. (Max addressable: 2 ^ 36) */
276 uint32_t pfn : 24;
277 /** The GVM handle. (127 VMs) */
278 uint32_t hGVM : 7;
279 /** The top page state bit, MBZ. */
280 uint32_t fZero : 1;
281 } Private;
282
283 /** The view of a shared page. */
284 struct GMMPAGESHARED
285 {
286 /** The reference count. */
287 uint32_t cRefs : 30;
288 /** The page state. */
289 uint32_t u2State : 2;
290 } Shared;
291
292 /** The view of a free page. */
293 struct GMMPAGEFREE
294 {
295 /** The index of the next page in the free list. UINT16_MAX is NIL. */
296 uint32_t iNext : 16;
297 /** Reserved. Checksum or something? */
298 uint32_t u14Reserved : 14;
299 /** The page state. */
300 uint32_t u2State : 2;
301 } Free;
302#endif
303} GMMPAGE;
304AssertCompileSize(GMMPAGE, sizeof(RTHCUINTPTR));
305/** Pointer to a GMMPAGE. */
306typedef GMMPAGE *PGMMPAGE;
307
308
309/** @name The Page States.
310 * @{ */
311/** A private page. */
312#define GMM_PAGE_STATE_PRIVATE 0
313/** A private page - alternative value used on the 32-bit implementation.
314 * This will never be used on 64-bit hosts. */
315#define GMM_PAGE_STATE_PRIVATE_32 1
316/** A shared page. */
317#define GMM_PAGE_STATE_SHARED 2
318/** A free page. */
319#define GMM_PAGE_STATE_FREE 3
320/** @} */
321
322
323/** @def GMM_PAGE_IS_PRIVATE
324 *
325 * @returns true if private, false if not.
326 * @param pPage The GMM page.
327 */
328#if HC_ARCH_BITS == 64
329# define GMM_PAGE_IS_PRIVATE(pPage) ( (pPage)->Common.u2State == GMM_PAGE_STATE_PRIVATE )
330#else
331# define GMM_PAGE_IS_PRIVATE(pPage) ( (pPage)->Private.fZero == 0 )
332#endif
333
334/** @def GMM_PAGE_IS_SHARED
335 *
336 * @returns true if shared, false if not.
337 * @param pPage The GMM page.
338 */
339#define GMM_PAGE_IS_SHARED(pPage) ( (pPage)->Common.u2State == GMM_PAGE_STATE_SHARED )
340
341/** @def GMM_PAGE_IS_FREE
342 *
343 * @returns true if free, false if not.
344 * @param pPage The GMM page.
345 */
346#define GMM_PAGE_IS_FREE(pPage) ( (pPage)->Common.u2State == GMM_PAGE_STATE_FREE )
347
348/** @def GMM_PAGE_PFN_LAST
349 * The last valid guest pfn range.
350 * @remark Some of the values outside the range has special meaning,
351 * see GMM_PAGE_PFN_UNSHAREABLE.
352 */
353#if HC_ARCH_BITS == 64
354# define GMM_PAGE_PFN_LAST UINT32_C(0xfffffff0)
355#else
356# define GMM_PAGE_PFN_LAST UINT32_C(0x00fffff0)
357#endif
358AssertCompile(GMM_PAGE_PFN_LAST == (GMM_GCPHYS_LAST >> PAGE_SHIFT));
359
360/** @def GMM_PAGE_PFN_UNSHAREABLE
361 * Indicates that this page isn't used for normal guest memory and thus isn't shareable.
362 */
363#if HC_ARCH_BITS == 64
364# define GMM_PAGE_PFN_UNSHAREABLE UINT32_C(0xfffffff1)
365#else
366# define GMM_PAGE_PFN_UNSHAREABLE UINT32_C(0x00fffff1)
367#endif
368AssertCompile(GMM_PAGE_PFN_UNSHAREABLE == (GMM_GCPHYS_UNSHAREABLE >> PAGE_SHIFT));
369
370
371/**
372 * A GMM allocation chunk ring-3 mapping record.
373 *
374 * This should really be associated with a session and not a VM, but
375 * it's simpler to associated with a VM and cleanup with the VM object
376 * is destroyed.
377 */
378typedef struct GMMCHUNKMAP
379{
380 /** The mapping object. */
381 RTR0MEMOBJ hMapObj;
382 /** The VM owning the mapping. */
383 PGVM pGVM;
384} GMMCHUNKMAP;
385/** Pointer to a GMM allocation chunk mapping. */
386typedef struct GMMCHUNKMAP *PGMMCHUNKMAP;
387
388
389/**
390 * A GMM allocation chunk.
391 */
392typedef struct GMMCHUNK
393{
394 /** The AVL node core.
395 * The Key is the chunk ID. (Giant mtx.) */
396 AVLU32NODECORE Core;
397 /** The memory object.
398 * Either from RTR0MemObjAllocPhysNC or RTR0MemObjLockUser depending on
399 * what the host can dish up with. (Chunk mtx protects mapping accesses
400 * and related frees.) */
401 RTR0MEMOBJ hMemObj;
402 /** Pointer to the next chunk in the free list. (Giant mtx.) */
403 PGMMCHUNK pFreeNext;
404 /** Pointer to the previous chunk in the free list. (Giant mtx.) */
405 PGMMCHUNK pFreePrev;
406 /** Pointer to the free set this chunk belongs to. NULL for
407 * chunks with no free pages. (Giant mtx.) */
408 PGMMCHUNKFREESET pSet;
409 /** List node in the chunk list (GMM::ChunkList). (Giant mtx.) */
410 RTLISTNODE ListNode;
411 /** Pointer to an array of mappings. (Chunk mtx.) */
412 PGMMCHUNKMAP paMappingsX;
413 /** The number of mappings. (Chunk mtx.) */
414 uint16_t cMappingsX;
415 /** The mapping lock this chunk is using using. UINT16_MAX if nobody is
416 * mapping or freeing anything. (Giant mtx.) */
417 uint8_t volatile iChunkMtx;
418 /** Flags field reserved for future use (like eliminating enmType).
419 * (Giant mtx.) */
420 uint8_t fFlags;
421 /** The head of the list of free pages. UINT16_MAX is the NIL value.
422 * (Giant mtx.) */
423 uint16_t iFreeHead;
424 /** The number of free pages. (Giant mtx.) */
425 uint16_t cFree;
426 /** The GVM handle of the VM that first allocated pages from this chunk, this
427 * is used as a preference when there are several chunks to choose from.
428 * When in bound memory mode this isn't a preference any longer. (Giant
429 * mtx.) */
430 uint16_t hGVM;
431 /** The ID of the NUMA node the memory mostly resides on. (Reserved for
432 * future use.) (Giant mtx.) */
433 uint16_t idNumaNode;
434 /** The number of private pages. (Giant mtx.) */
435 uint16_t cPrivate;
436 /** The number of shared pages. (Giant mtx.) */
437 uint16_t cShared;
438 /** The pages. (Giant mtx.) */
439 GMMPAGE aPages[GMM_CHUNK_SIZE >> PAGE_SHIFT];
440} GMMCHUNK;
441
442/** Indicates that the NUMA properies of the memory is unknown. */
443#define GMM_CHUNK_NUMA_ID_UNKNOWN UINT16_C(0xfffe)
444
445/** @name GMM_CHUNK_FLAGS_XXX - chunk flags.
446 * @{ */
447/** Indicates that the chunk is a large page (2MB). */
448#define GMM_CHUNK_FLAGS_LARGE_PAGE UINT16_C(0x0001)
449/** @} */
450
451
452/**
453 * An allocation chunk TLB entry.
454 */
455typedef struct GMMCHUNKTLBE
456{
457 /** The chunk id. */
458 uint32_t idChunk;
459 /** Pointer to the chunk. */
460 PGMMCHUNK pChunk;
461} GMMCHUNKTLBE;
462/** Pointer to an allocation chunk TLB entry. */
463typedef GMMCHUNKTLBE *PGMMCHUNKTLBE;
464
465
466/** The number of entries tin the allocation chunk TLB. */
467#define GMM_CHUNKTLB_ENTRIES 32
468/** Gets the TLB entry index for the given Chunk ID. */
469#define GMM_CHUNKTLB_IDX(idChunk) ( (idChunk) & (GMM_CHUNKTLB_ENTRIES - 1) )
470
471/**
472 * An allocation chunk TLB.
473 */
474typedef struct GMMCHUNKTLB
475{
476 /** The TLB entries. */
477 GMMCHUNKTLBE aEntries[GMM_CHUNKTLB_ENTRIES];
478} GMMCHUNKTLB;
479/** Pointer to an allocation chunk TLB. */
480typedef GMMCHUNKTLB *PGMMCHUNKTLB;
481
482
483/**
484 * The GMM instance data.
485 */
486typedef struct GMM
487{
488 /** Magic / eye catcher. GMM_MAGIC */
489 uint32_t u32Magic;
490 /** The number of threads waiting on the mutex. */
491 uint32_t cMtxContenders;
492#ifdef VBOX_USE_CRIT_SECT_FOR_GIANT
493 /** The critical section protecting the GMM.
494 * More fine grained locking can be implemented later if necessary. */
495 RTCRITSECT GiantCritSect;
496#else
497 /** The fast mutex protecting the GMM.
498 * More fine grained locking can be implemented later if necessary. */
499 RTSEMFASTMUTEX hMtx;
500#endif
501#ifdef VBOX_STRICT
502 /** The current mutex owner. */
503 RTNATIVETHREAD hMtxOwner;
504#endif
505 /** The chunk tree. */
506 PAVLU32NODECORE pChunks;
507 /** The chunk TLB. */
508 GMMCHUNKTLB ChunkTLB;
509 /** The private free set. */
510 GMMCHUNKFREESET PrivateX;
511 /** The shared free set. */
512 GMMCHUNKFREESET Shared;
513
514 /** Shared module tree (global).
515 * @todo separate trees for distinctly different guest OSes. */
516 PAVLLU32NODECORE pGlobalSharedModuleTree;
517 /** Sharable modules (count of nodes in pGlobalSharedModuleTree). */
518 uint32_t cShareableModules;
519
520 /** The chunk list. For simplifying the cleanup process. */
521 RTLISTANCHOR ChunkList;
522
523 /** The maximum number of pages we're allowed to allocate.
524 * @gcfgm{GMM/MaxPages,64-bit, Direct.}
525 * @gcfgm{GMM/PctPages,32-bit, Relative to the number of host pages.} */
526 uint64_t cMaxPages;
527 /** The number of pages that has been reserved.
528 * The deal is that cReservedPages - cOverCommittedPages <= cMaxPages. */
529 uint64_t cReservedPages;
530 /** The number of pages that we have over-committed in reservations. */
531 uint64_t cOverCommittedPages;
532 /** The number of actually allocated (committed if you like) pages. */
533 uint64_t cAllocatedPages;
534 /** The number of pages that are shared. A subset of cAllocatedPages. */
535 uint64_t cSharedPages;
536 /** The number of pages that are actually shared between VMs. */
537 uint64_t cDuplicatePages;
538 /** The number of pages that are shared that has been left behind by
539 * VMs not doing proper cleanups. */
540 uint64_t cLeftBehindSharedPages;
541 /** The number of allocation chunks.
542 * (The number of pages we've allocated from the host can be derived from this.) */
543 uint32_t cChunks;
544 /** The number of current ballooned pages. */
545 uint64_t cBalloonedPages;
546
547 /** The legacy allocation mode indicator.
548 * This is determined at initialization time. */
549 bool fLegacyAllocationMode;
550 /** The bound memory mode indicator.
551 * When set, the memory will be bound to a specific VM and never
552 * shared. This is always set if fLegacyAllocationMode is set.
553 * (Also determined at initialization time.) */
554 bool fBoundMemoryMode;
555 /** The number of registered VMs. */
556 uint16_t cRegisteredVMs;
557
558 /** The number of freed chunks ever. This is used a list generation to
559 * avoid restarting the cleanup scanning when the list wasn't modified. */
560 uint32_t cFreedChunks;
561 /** The previous allocated Chunk ID.
562 * Used as a hint to avoid scanning the whole bitmap. */
563 uint32_t idChunkPrev;
564 /** Chunk ID allocation bitmap.
565 * Bits of allocated IDs are set, free ones are clear.
566 * The NIL id (0) is marked allocated. */
567 uint32_t bmChunkId[(GMM_CHUNKID_LAST + 1 + 31) / 32];
568
569 /** The index of the next mutex to use. */
570 uint32_t iNextChunkMtx;
571 /** Chunk locks for reducing lock contention without having to allocate
572 * one lock per chunk. */
573 struct
574 {
575 /** The mutex */
576 RTSEMFASTMUTEX hMtx;
577 /** The number of threads currently using this mutex. */
578 uint32_t volatile cUsers;
579 } aChunkMtx[64];
580} GMM;
581/** Pointer to the GMM instance. */
582typedef GMM *PGMM;
583
584/** The value of GMM::u32Magic (Katsuhiro Otomo). */
585#define GMM_MAGIC UINT32_C(0x19540414)
586
587
588/**
589 * GMM chunk mutex state.
590 *
591 * This is returned by gmmR0ChunkMutexAcquire and is used by the other
592 * gmmR0ChunkMutex* methods.
593 */
594typedef struct GMMR0CHUNKMTXSTATE
595{
596 PGMM pGMM;
597 /** The index of the chunk mutex. */
598 uint8_t iChunkMtx;
599 /** The relevant flags (GMMR0CHUNK_MTX_XXX). */
600 uint8_t fFlags;
601} GMMR0CHUNKMTXSTATE;
602/** Pointer to a chunk mutex state. */
603typedef GMMR0CHUNKMTXSTATE *PGMMR0CHUNKMTXSTATE;
604
605/** @name GMMR0CHUNK_MTX_XXX
606 * @{ */
607#define GMMR0CHUNK_MTX_INVALID UINT32_C(0)
608#define GMMR0CHUNK_MTX_KEEP_GIANT UINT32_C(1)
609#define GMMR0CHUNK_MTX_RETAKE_GIANT UINT32_C(2)
610#define GMMR0CHUNK_MTX_DROP_GIANT UINT32_C(3)
611#define GMMR0CHUNK_MTX_END UINT32_C(4)
612/** @} */
613
614
615/** The maximum number of shared modules per-vm. */
616#define GMM_MAX_SHARED_PER_VM_MODULES 2048
617/** The maximum number of shared modules GMM is allowed to track. */
618#define GMM_MAX_SHARED_GLOBAL_MODULES 16834
619
620
621/**
622 * Argument packet for gmmR0SharedModuleCleanup.
623 */
624typedef struct GMMR0SHMODPERVMDTORARGS
625{
626 PGVM pGVM;
627 PGMM pGMM;
628} GMMR0SHMODPERVMDTORARGS;
629
630/**
631 * Argument packet for gmmR0CheckSharedModule.
632 */
633typedef struct GMMCHECKSHAREDMODULEINFO
634{
635 PGVM pGVM;
636 VMCPUID idCpu;
637} GMMCHECKSHAREDMODULEINFO;
638
639/**
640 * Argument packet for gmmR0FindDupPageInChunk by GMMR0FindDuplicatePage.
641 */
642typedef struct GMMFINDDUPPAGEINFO
643{
644 PGVM pGVM;
645 PGMM pGMM;
646 uint8_t *pSourcePage;
647 bool fFoundDuplicate;
648} GMMFINDDUPPAGEINFO;
649
650
651/*********************************************************************************************************************************
652* Global Variables *
653*********************************************************************************************************************************/
654/** Pointer to the GMM instance data. */
655static PGMM g_pGMM = NULL;
656
657/** Macro for obtaining and validating the g_pGMM pointer.
658 *
659 * On failure it will return from the invoking function with the specified
660 * return value.
661 *
662 * @param pGMM The name of the pGMM variable.
663 * @param rc The return value on failure. Use VERR_GMM_INSTANCE for VBox
664 * status codes.
665 */
666#define GMM_GET_VALID_INSTANCE(pGMM, rc) \
667 do { \
668 (pGMM) = g_pGMM; \
669 AssertPtrReturn((pGMM), (rc)); \
670 AssertMsgReturn((pGMM)->u32Magic == GMM_MAGIC, ("%p - %#x\n", (pGMM), (pGMM)->u32Magic), (rc)); \
671 } while (0)
672
673/** Macro for obtaining and validating the g_pGMM pointer, void function
674 * variant.
675 *
676 * On failure it will return from the invoking function.
677 *
678 * @param pGMM The name of the pGMM variable.
679 */
680#define GMM_GET_VALID_INSTANCE_VOID(pGMM) \
681 do { \
682 (pGMM) = g_pGMM; \
683 AssertPtrReturnVoid((pGMM)); \
684 AssertMsgReturnVoid((pGMM)->u32Magic == GMM_MAGIC, ("%p - %#x\n", (pGMM), (pGMM)->u32Magic)); \
685 } while (0)
686
687
688/** @def GMM_CHECK_SANITY_UPON_ENTERING
689 * Checks the sanity of the GMM instance data before making changes.
690 *
691 * This is macro is a stub by default and must be enabled manually in the code.
692 *
693 * @returns true if sane, false if not.
694 * @param pGMM The name of the pGMM variable.
695 */
696#if defined(VBOX_STRICT) && defined(GMMR0_WITH_SANITY_CHECK) && 0
697# define GMM_CHECK_SANITY_UPON_ENTERING(pGMM) (gmmR0SanityCheck((pGMM), __PRETTY_FUNCTION__, __LINE__) == 0)
698#else
699# define GMM_CHECK_SANITY_UPON_ENTERING(pGMM) (true)
700#endif
701
702/** @def GMM_CHECK_SANITY_UPON_LEAVING
703 * Checks the sanity of the GMM instance data after making changes.
704 *
705 * This is macro is a stub by default and must be enabled manually in the code.
706 *
707 * @returns true if sane, false if not.
708 * @param pGMM The name of the pGMM variable.
709 */
710#if defined(VBOX_STRICT) && defined(GMMR0_WITH_SANITY_CHECK) && 0
711# define GMM_CHECK_SANITY_UPON_LEAVING(pGMM) (gmmR0SanityCheck((pGMM), __PRETTY_FUNCTION__, __LINE__) == 0)
712#else
713# define GMM_CHECK_SANITY_UPON_LEAVING(pGMM) (true)
714#endif
715
716/** @def GMM_CHECK_SANITY_IN_LOOPS
717 * Checks the sanity of the GMM instance in the allocation loops.
718 *
719 * This is macro is a stub by default and must be enabled manually in the code.
720 *
721 * @returns true if sane, false if not.
722 * @param pGMM The name of the pGMM variable.
723 */
724#if defined(VBOX_STRICT) && defined(GMMR0_WITH_SANITY_CHECK) && 0
725# define GMM_CHECK_SANITY_IN_LOOPS(pGMM) (gmmR0SanityCheck((pGMM), __PRETTY_FUNCTION__, __LINE__) == 0)
726#else
727# define GMM_CHECK_SANITY_IN_LOOPS(pGMM) (true)
728#endif
729
730
731/*********************************************************************************************************************************
732* Internal Functions *
733*********************************************************************************************************************************/
734static DECLCALLBACK(int) gmmR0TermDestroyChunk(PAVLU32NODECORE pNode, void *pvGMM);
735static bool gmmR0CleanupVMScanChunk(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk);
736DECLINLINE(void) gmmR0UnlinkChunk(PGMMCHUNK pChunk);
737DECLINLINE(void) gmmR0LinkChunk(PGMMCHUNK pChunk, PGMMCHUNKFREESET pSet);
738DECLINLINE(void) gmmR0SelectSetAndLinkChunk(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk);
739#ifdef GMMR0_WITH_SANITY_CHECK
740static uint32_t gmmR0SanityCheck(PGMM pGMM, const char *pszFunction, unsigned uLineNo);
741#endif
742static bool gmmR0FreeChunk(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk, bool fRelaxedSem);
743DECLINLINE(void) gmmR0FreePrivatePage(PGMM pGMM, PGVM pGVM, uint32_t idPage, PGMMPAGE pPage);
744DECLINLINE(void) gmmR0FreeSharedPage(PGMM pGMM, PGVM pGVM, uint32_t idPage, PGMMPAGE pPage);
745static int gmmR0UnmapChunkLocked(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk);
746#ifdef VBOX_WITH_PAGE_SHARING
747static void gmmR0SharedModuleCleanup(PGMM pGMM, PGVM pGVM);
748# ifdef VBOX_STRICT
749static uint32_t gmmR0StrictPageChecksum(PGMM pGMM, PGVM pGVM, uint32_t idPage);
750# endif
751#endif
752
753
754
755/**
756 * Initializes the GMM component.
757 *
758 * This is called when the VMMR0.r0 module is loaded and protected by the
759 * loader semaphore.
760 *
761 * @returns VBox status code.
762 */
763GMMR0DECL(int) GMMR0Init(void)
764{
765 LogFlow(("GMMInit:\n"));
766
767 /*
768 * Allocate the instance data and the locks.
769 */
770 PGMM pGMM = (PGMM)RTMemAllocZ(sizeof(*pGMM));
771 if (!pGMM)
772 return VERR_NO_MEMORY;
773
774 pGMM->u32Magic = GMM_MAGIC;
775 for (unsigned i = 0; i < RT_ELEMENTS(pGMM->ChunkTLB.aEntries); i++)
776 pGMM->ChunkTLB.aEntries[i].idChunk = NIL_GMM_CHUNKID;
777 RTListInit(&pGMM->ChunkList);
778 ASMBitSet(&pGMM->bmChunkId[0], NIL_GMM_CHUNKID);
779
780#ifdef VBOX_USE_CRIT_SECT_FOR_GIANT
781 int rc = RTCritSectInit(&pGMM->GiantCritSect);
782#else
783 int rc = RTSemFastMutexCreate(&pGMM->hMtx);
784#endif
785 if (RT_SUCCESS(rc))
786 {
787 unsigned iMtx;
788 for (iMtx = 0; iMtx < RT_ELEMENTS(pGMM->aChunkMtx); iMtx++)
789 {
790 rc = RTSemFastMutexCreate(&pGMM->aChunkMtx[iMtx].hMtx);
791 if (RT_FAILURE(rc))
792 break;
793 }
794 if (RT_SUCCESS(rc))
795 {
796 /*
797 * Check and see if RTR0MemObjAllocPhysNC works.
798 */
799#if 0 /* later, see @bufref{3170}. */
800 RTR0MEMOBJ MemObj;
801 rc = RTR0MemObjAllocPhysNC(&MemObj, _64K, NIL_RTHCPHYS);
802 if (RT_SUCCESS(rc))
803 {
804 rc = RTR0MemObjFree(MemObj, true);
805 AssertRC(rc);
806 }
807 else if (rc == VERR_NOT_SUPPORTED)
808 pGMM->fLegacyAllocationMode = pGMM->fBoundMemoryMode = true;
809 else
810 SUPR0Printf("GMMR0Init: RTR0MemObjAllocPhysNC(,64K,Any) -> %d!\n", rc);
811#else
812# if defined(RT_OS_WINDOWS) || (defined(RT_OS_SOLARIS) && ARCH_BITS == 64) || defined(RT_OS_LINUX) || defined(RT_OS_FREEBSD)
813 pGMM->fLegacyAllocationMode = false;
814# if ARCH_BITS == 32
815 /* Don't reuse possibly partial chunks because of the virtual
816 address space limitation. */
817 pGMM->fBoundMemoryMode = true;
818# else
819 pGMM->fBoundMemoryMode = false;
820# endif
821# else
822 pGMM->fLegacyAllocationMode = true;
823 pGMM->fBoundMemoryMode = true;
824# endif
825#endif
826
827 /*
828 * Query system page count and guess a reasonable cMaxPages value.
829 */
830 pGMM->cMaxPages = UINT32_MAX; /** @todo IPRT function for query ram size and such. */
831
832 g_pGMM = pGMM;
833 LogFlow(("GMMInit: pGMM=%p fLegacyAllocationMode=%RTbool fBoundMemoryMode=%RTbool\n", pGMM, pGMM->fLegacyAllocationMode, pGMM->fBoundMemoryMode));
834 return VINF_SUCCESS;
835 }
836
837 /*
838 * Bail out.
839 */
840 while (iMtx-- > 0)
841 RTSemFastMutexDestroy(pGMM->aChunkMtx[iMtx].hMtx);
842#ifdef VBOX_USE_CRIT_SECT_FOR_GIANT
843 RTCritSectDelete(&pGMM->GiantCritSect);
844#else
845 RTSemFastMutexDestroy(pGMM->hMtx);
846#endif
847 }
848
849 pGMM->u32Magic = 0;
850 RTMemFree(pGMM);
851 SUPR0Printf("GMMR0Init: failed! rc=%d\n", rc);
852 return rc;
853}
854
855
856/**
857 * Terminates the GMM component.
858 */
859GMMR0DECL(void) GMMR0Term(void)
860{
861 LogFlow(("GMMTerm:\n"));
862
863 /*
864 * Take care / be paranoid...
865 */
866 PGMM pGMM = g_pGMM;
867 if (!VALID_PTR(pGMM))
868 return;
869 if (pGMM->u32Magic != GMM_MAGIC)
870 {
871 SUPR0Printf("GMMR0Term: u32Magic=%#x\n", pGMM->u32Magic);
872 return;
873 }
874
875 /*
876 * Undo what init did and free all the resources we've acquired.
877 */
878 /* Destroy the fundamentals. */
879 g_pGMM = NULL;
880 pGMM->u32Magic = ~GMM_MAGIC;
881#ifdef VBOX_USE_CRIT_SECT_FOR_GIANT
882 RTCritSectDelete(&pGMM->GiantCritSect);
883#else
884 RTSemFastMutexDestroy(pGMM->hMtx);
885 pGMM->hMtx = NIL_RTSEMFASTMUTEX;
886#endif
887
888 /* Free any chunks still hanging around. */
889 RTAvlU32Destroy(&pGMM->pChunks, gmmR0TermDestroyChunk, pGMM);
890
891 /* Destroy the chunk locks. */
892 for (unsigned iMtx = 0; iMtx < RT_ELEMENTS(pGMM->aChunkMtx); iMtx++)
893 {
894 Assert(pGMM->aChunkMtx[iMtx].cUsers == 0);
895 RTSemFastMutexDestroy(pGMM->aChunkMtx[iMtx].hMtx);
896 pGMM->aChunkMtx[iMtx].hMtx = NIL_RTSEMFASTMUTEX;
897 }
898
899 /* Finally the instance data itself. */
900 RTMemFree(pGMM);
901 LogFlow(("GMMTerm: done\n"));
902}
903
904
905/**
906 * RTAvlU32Destroy callback.
907 *
908 * @returns 0
909 * @param pNode The node to destroy.
910 * @param pvGMM The GMM handle.
911 */
912static DECLCALLBACK(int) gmmR0TermDestroyChunk(PAVLU32NODECORE pNode, void *pvGMM)
913{
914 PGMMCHUNK pChunk = (PGMMCHUNK)pNode;
915
916 if (pChunk->cFree != (GMM_CHUNK_SIZE >> PAGE_SHIFT))
917 SUPR0Printf("GMMR0Term: %p/%#x: cFree=%d cPrivate=%d cShared=%d cMappings=%d\n", pChunk,
918 pChunk->Core.Key, pChunk->cFree, pChunk->cPrivate, pChunk->cShared, pChunk->cMappingsX);
919
920 int rc = RTR0MemObjFree(pChunk->hMemObj, true /* fFreeMappings */);
921 if (RT_FAILURE(rc))
922 {
923 SUPR0Printf("GMMR0Term: %p/%#x: RTRMemObjFree(%p,true) -> %d (cMappings=%d)\n", pChunk,
924 pChunk->Core.Key, pChunk->hMemObj, rc, pChunk->cMappingsX);
925 AssertRC(rc);
926 }
927 pChunk->hMemObj = NIL_RTR0MEMOBJ;
928
929 RTMemFree(pChunk->paMappingsX);
930 pChunk->paMappingsX = NULL;
931
932 RTMemFree(pChunk);
933 NOREF(pvGMM);
934 return 0;
935}
936
937
938/**
939 * Initializes the per-VM data for the GMM.
940 *
941 * This is called from within the GVMM lock (from GVMMR0CreateVM)
942 * and should only initialize the data members so GMMR0CleanupVM
943 * can deal with them. We reserve no memory or anything here,
944 * that's done later in GMMR0InitVM.
945 *
946 * @param pGVM Pointer to the Global VM structure.
947 */
948GMMR0DECL(void) GMMR0InitPerVMData(PGVM pGVM)
949{
950 AssertCompile(RT_SIZEOFMEMB(GVM,gmm.s) <= RT_SIZEOFMEMB(GVM,gmm.padding));
951
952 pGVM->gmm.s.Stats.enmPolicy = GMMOCPOLICY_INVALID;
953 pGVM->gmm.s.Stats.enmPriority = GMMPRIORITY_INVALID;
954 pGVM->gmm.s.Stats.fMayAllocate = false;
955}
956
957
958/**
959 * Acquires the GMM giant lock.
960 *
961 * @returns Assert status code from RTSemFastMutexRequest.
962 * @param pGMM Pointer to the GMM instance.
963 */
964static int gmmR0MutexAcquire(PGMM pGMM)
965{
966 ASMAtomicIncU32(&pGMM->cMtxContenders);
967#ifdef VBOX_USE_CRIT_SECT_FOR_GIANT
968 int rc = RTCritSectEnter(&pGMM->GiantCritSect);
969#else
970 int rc = RTSemFastMutexRequest(pGMM->hMtx);
971#endif
972 ASMAtomicDecU32(&pGMM->cMtxContenders);
973 AssertRC(rc);
974#ifdef VBOX_STRICT
975 pGMM->hMtxOwner = RTThreadNativeSelf();
976#endif
977 return rc;
978}
979
980
981/**
982 * Releases the GMM giant lock.
983 *
984 * @returns Assert status code from RTSemFastMutexRequest.
985 * @param pGMM Pointer to the GMM instance.
986 */
987static int gmmR0MutexRelease(PGMM pGMM)
988{
989#ifdef VBOX_STRICT
990 pGMM->hMtxOwner = NIL_RTNATIVETHREAD;
991#endif
992#ifdef VBOX_USE_CRIT_SECT_FOR_GIANT
993 int rc = RTCritSectLeave(&pGMM->GiantCritSect);
994#else
995 int rc = RTSemFastMutexRelease(pGMM->hMtx);
996 AssertRC(rc);
997#endif
998 return rc;
999}
1000
1001
1002/**
1003 * Yields the GMM giant lock if there is contention and a certain minimum time
1004 * has elapsed since we took it.
1005 *
1006 * @returns @c true if the mutex was yielded, @c false if not.
1007 * @param pGMM Pointer to the GMM instance.
1008 * @param puLockNanoTS Where the lock acquisition time stamp is kept
1009 * (in/out).
1010 */
1011static bool gmmR0MutexYield(PGMM pGMM, uint64_t *puLockNanoTS)
1012{
1013 /*
1014 * If nobody is contending the mutex, don't bother checking the time.
1015 */
1016 if (ASMAtomicReadU32(&pGMM->cMtxContenders) == 0)
1017 return false;
1018
1019 /*
1020 * Don't yield if we haven't executed for at least 2 milliseconds.
1021 */
1022 uint64_t uNanoNow = RTTimeSystemNanoTS();
1023 if (uNanoNow - *puLockNanoTS < UINT32_C(2000000))
1024 return false;
1025
1026 /*
1027 * Yield the mutex.
1028 */
1029#ifdef VBOX_STRICT
1030 pGMM->hMtxOwner = NIL_RTNATIVETHREAD;
1031#endif
1032 ASMAtomicIncU32(&pGMM->cMtxContenders);
1033#ifdef VBOX_USE_CRIT_SECT_FOR_GIANT
1034 int rc1 = RTCritSectLeave(&pGMM->GiantCritSect); AssertRC(rc1);
1035#else
1036 int rc1 = RTSemFastMutexRelease(pGMM->hMtx); AssertRC(rc1);
1037#endif
1038
1039 RTThreadYield();
1040
1041#ifdef VBOX_USE_CRIT_SECT_FOR_GIANT
1042 int rc2 = RTCritSectEnter(&pGMM->GiantCritSect); AssertRC(rc2);
1043#else
1044 int rc2 = RTSemFastMutexRequest(pGMM->hMtx); AssertRC(rc2);
1045#endif
1046 *puLockNanoTS = RTTimeSystemNanoTS();
1047 ASMAtomicDecU32(&pGMM->cMtxContenders);
1048#ifdef VBOX_STRICT
1049 pGMM->hMtxOwner = RTThreadNativeSelf();
1050#endif
1051
1052 return true;
1053}
1054
1055
1056/**
1057 * Acquires a chunk lock.
1058 *
1059 * The caller must own the giant lock.
1060 *
1061 * @returns Assert status code from RTSemFastMutexRequest.
1062 * @param pMtxState The chunk mutex state info. (Avoids
1063 * passing the same flags and stuff around
1064 * for subsequent release and drop-giant
1065 * calls.)
1066 * @param pGMM Pointer to the GMM instance.
1067 * @param pChunk Pointer to the chunk.
1068 * @param fFlags Flags regarding the giant lock, GMMR0CHUNK_MTX_XXX.
1069 */
1070static int gmmR0ChunkMutexAcquire(PGMMR0CHUNKMTXSTATE pMtxState, PGMM pGMM, PGMMCHUNK pChunk, uint32_t fFlags)
1071{
1072 Assert(fFlags > GMMR0CHUNK_MTX_INVALID && fFlags < GMMR0CHUNK_MTX_END);
1073 Assert(pGMM->hMtxOwner == RTThreadNativeSelf());
1074
1075 pMtxState->pGMM = pGMM;
1076 pMtxState->fFlags = (uint8_t)fFlags;
1077
1078 /*
1079 * Get the lock index and reference the lock.
1080 */
1081 Assert(pGMM->hMtxOwner == RTThreadNativeSelf());
1082 uint32_t iChunkMtx = pChunk->iChunkMtx;
1083 if (iChunkMtx == UINT8_MAX)
1084 {
1085 iChunkMtx = pGMM->iNextChunkMtx++;
1086 iChunkMtx %= RT_ELEMENTS(pGMM->aChunkMtx);
1087
1088 /* Try get an unused one... */
1089 if (pGMM->aChunkMtx[iChunkMtx].cUsers)
1090 {
1091 iChunkMtx = pGMM->iNextChunkMtx++;
1092 iChunkMtx %= RT_ELEMENTS(pGMM->aChunkMtx);
1093 if (pGMM->aChunkMtx[iChunkMtx].cUsers)
1094 {
1095 iChunkMtx = pGMM->iNextChunkMtx++;
1096 iChunkMtx %= RT_ELEMENTS(pGMM->aChunkMtx);
1097 if (pGMM->aChunkMtx[iChunkMtx].cUsers)
1098 {
1099 iChunkMtx = pGMM->iNextChunkMtx++;
1100 iChunkMtx %= RT_ELEMENTS(pGMM->aChunkMtx);
1101 }
1102 }
1103 }
1104
1105 pChunk->iChunkMtx = iChunkMtx;
1106 }
1107 AssertCompile(RT_ELEMENTS(pGMM->aChunkMtx) < UINT8_MAX);
1108 pMtxState->iChunkMtx = (uint8_t)iChunkMtx;
1109 ASMAtomicIncU32(&pGMM->aChunkMtx[iChunkMtx].cUsers);
1110
1111 /*
1112 * Drop the giant?
1113 */
1114 if (fFlags != GMMR0CHUNK_MTX_KEEP_GIANT)
1115 {
1116 /** @todo GMM life cycle cleanup (we may race someone
1117 * destroying and cleaning up GMM)? */
1118 gmmR0MutexRelease(pGMM);
1119 }
1120
1121 /*
1122 * Take the chunk mutex.
1123 */
1124 int rc = RTSemFastMutexRequest(pGMM->aChunkMtx[iChunkMtx].hMtx);
1125 AssertRC(rc);
1126 return rc;
1127}
1128
1129
1130/**
1131 * Releases the GMM giant lock.
1132 *
1133 * @returns Assert status code from RTSemFastMutexRequest.
1134 * @param pMtxState Pointer to the chunk mutex state.
1135 * @param pChunk Pointer to the chunk if it's still
1136 * alive, NULL if it isn't. This is used to deassociate
1137 * the chunk from the mutex on the way out so a new one
1138 * can be selected next time, thus avoiding contented
1139 * mutexes.
1140 */
1141static int gmmR0ChunkMutexRelease(PGMMR0CHUNKMTXSTATE pMtxState, PGMMCHUNK pChunk)
1142{
1143 PGMM pGMM = pMtxState->pGMM;
1144
1145 /*
1146 * Release the chunk mutex and reacquire the giant if requested.
1147 */
1148 int rc = RTSemFastMutexRelease(pGMM->aChunkMtx[pMtxState->iChunkMtx].hMtx);
1149 AssertRC(rc);
1150 if (pMtxState->fFlags == GMMR0CHUNK_MTX_RETAKE_GIANT)
1151 rc = gmmR0MutexAcquire(pGMM);
1152 else
1153 Assert((pMtxState->fFlags != GMMR0CHUNK_MTX_DROP_GIANT) == (pGMM->hMtxOwner == RTThreadNativeSelf()));
1154
1155 /*
1156 * Drop the chunk mutex user reference and deassociate it from the chunk
1157 * when possible.
1158 */
1159 if ( ASMAtomicDecU32(&pGMM->aChunkMtx[pMtxState->iChunkMtx].cUsers) == 0
1160 && pChunk
1161 && RT_SUCCESS(rc) )
1162 {
1163 if (pMtxState->fFlags != GMMR0CHUNK_MTX_DROP_GIANT)
1164 pChunk->iChunkMtx = UINT8_MAX;
1165 else
1166 {
1167 rc = gmmR0MutexAcquire(pGMM);
1168 if (RT_SUCCESS(rc))
1169 {
1170 if (pGMM->aChunkMtx[pMtxState->iChunkMtx].cUsers == 0)
1171 pChunk->iChunkMtx = UINT8_MAX;
1172 rc = gmmR0MutexRelease(pGMM);
1173 }
1174 }
1175 }
1176
1177 pMtxState->pGMM = NULL;
1178 return rc;
1179}
1180
1181
1182/**
1183 * Drops the giant GMM lock we kept in gmmR0ChunkMutexAcquire while keeping the
1184 * chunk locked.
1185 *
1186 * This only works if gmmR0ChunkMutexAcquire was called with
1187 * GMMR0CHUNK_MTX_KEEP_GIANT. gmmR0ChunkMutexRelease will retake the giant
1188 * mutex, i.e. behave as if GMMR0CHUNK_MTX_RETAKE_GIANT was used.
1189 *
1190 * @returns VBox status code (assuming success is ok).
1191 * @param pMtxState Pointer to the chunk mutex state.
1192 */
1193static int gmmR0ChunkMutexDropGiant(PGMMR0CHUNKMTXSTATE pMtxState)
1194{
1195 AssertReturn(pMtxState->fFlags == GMMR0CHUNK_MTX_KEEP_GIANT, VERR_GMM_MTX_FLAGS);
1196 Assert(pMtxState->pGMM->hMtxOwner == RTThreadNativeSelf());
1197 pMtxState->fFlags = GMMR0CHUNK_MTX_RETAKE_GIANT;
1198 /** @todo GMM life cycle cleanup (we may race someone
1199 * destroying and cleaning up GMM)? */
1200 return gmmR0MutexRelease(pMtxState->pGMM);
1201}
1202
1203
1204/**
1205 * For experimenting with NUMA affinity and such.
1206 *
1207 * @returns The current NUMA Node ID.
1208 */
1209static uint16_t gmmR0GetCurrentNumaNodeId(void)
1210{
1211#if 1
1212 return GMM_CHUNK_NUMA_ID_UNKNOWN;
1213#else
1214 return RTMpCpuId() / 16;
1215#endif
1216}
1217
1218
1219
1220/**
1221 * Cleans up when a VM is terminating.
1222 *
1223 * @param pGVM Pointer to the Global VM structure.
1224 */
1225GMMR0DECL(void) GMMR0CleanupVM(PGVM pGVM)
1226{
1227 LogFlow(("GMMR0CleanupVM: pGVM=%p:{.pVM=%p, .hSelf=%#x}\n", pGVM, pGVM->pVM, pGVM->hSelf));
1228
1229 PGMM pGMM;
1230 GMM_GET_VALID_INSTANCE_VOID(pGMM);
1231
1232#ifdef VBOX_WITH_PAGE_SHARING
1233 /*
1234 * Clean up all registered shared modules first.
1235 */
1236 gmmR0SharedModuleCleanup(pGMM, pGVM);
1237#endif
1238
1239 gmmR0MutexAcquire(pGMM);
1240 uint64_t uLockNanoTS = RTTimeSystemNanoTS();
1241 GMM_CHECK_SANITY_UPON_ENTERING(pGMM);
1242
1243 /*
1244 * The policy is 'INVALID' until the initial reservation
1245 * request has been serviced.
1246 */
1247 if ( pGVM->gmm.s.Stats.enmPolicy > GMMOCPOLICY_INVALID
1248 && pGVM->gmm.s.Stats.enmPolicy < GMMOCPOLICY_END)
1249 {
1250 /*
1251 * If it's the last VM around, we can skip walking all the chunk looking
1252 * for the pages owned by this VM and instead flush the whole shebang.
1253 *
1254 * This takes care of the eventuality that a VM has left shared page
1255 * references behind (shouldn't happen of course, but you never know).
1256 */
1257 Assert(pGMM->cRegisteredVMs);
1258 pGMM->cRegisteredVMs--;
1259
1260 /*
1261 * Walk the entire pool looking for pages that belong to this VM
1262 * and leftover mappings. (This'll only catch private pages,
1263 * shared pages will be 'left behind'.)
1264 */
1265 /** @todo r=bird: This scanning+freeing could be optimized in bound mode! */
1266 uint64_t cPrivatePages = pGVM->gmm.s.Stats.cPrivatePages; /* save */
1267
1268 unsigned iCountDown = 64;
1269 bool fRedoFromStart;
1270 PGMMCHUNK pChunk;
1271 do
1272 {
1273 fRedoFromStart = false;
1274 RTListForEachReverse(&pGMM->ChunkList, pChunk, GMMCHUNK, ListNode)
1275 {
1276 uint32_t const cFreeChunksOld = pGMM->cFreedChunks;
1277 if ( ( !pGMM->fBoundMemoryMode
1278 || pChunk->hGVM == pGVM->hSelf)
1279 && gmmR0CleanupVMScanChunk(pGMM, pGVM, pChunk))
1280 {
1281 /* We left the giant mutex, so reset the yield counters. */
1282 uLockNanoTS = RTTimeSystemNanoTS();
1283 iCountDown = 64;
1284 }
1285 else
1286 {
1287 /* Didn't leave it, so do normal yielding. */
1288 if (!iCountDown)
1289 gmmR0MutexYield(pGMM, &uLockNanoTS);
1290 else
1291 iCountDown--;
1292 }
1293 if (pGMM->cFreedChunks != cFreeChunksOld)
1294 {
1295 fRedoFromStart = true;
1296 break;
1297 }
1298 }
1299 } while (fRedoFromStart);
1300
1301 if (pGVM->gmm.s.Stats.cPrivatePages)
1302 SUPR0Printf("GMMR0CleanupVM: hGVM=%#x has %#x private pages that cannot be found!\n", pGVM->hSelf, pGVM->gmm.s.Stats.cPrivatePages);
1303
1304 pGMM->cAllocatedPages -= cPrivatePages;
1305
1306 /*
1307 * Free empty chunks.
1308 */
1309 PGMMCHUNKFREESET pPrivateSet = pGMM->fBoundMemoryMode ? &pGVM->gmm.s.Private : &pGMM->PrivateX;
1310 do
1311 {
1312 fRedoFromStart = false;
1313 iCountDown = 10240;
1314 pChunk = pPrivateSet->apLists[GMM_CHUNK_FREE_SET_UNUSED_LIST];
1315 while (pChunk)
1316 {
1317 PGMMCHUNK pNext = pChunk->pFreeNext;
1318 Assert(pChunk->cFree == GMM_CHUNK_NUM_PAGES);
1319 if ( !pGMM->fBoundMemoryMode
1320 || pChunk->hGVM == pGVM->hSelf)
1321 {
1322 uint64_t const idGenerationOld = pPrivateSet->idGeneration;
1323 if (gmmR0FreeChunk(pGMM, pGVM, pChunk, true /*fRelaxedSem*/))
1324 {
1325 /* We've left the giant mutex, restart? (+1 for our unlink) */
1326 fRedoFromStart = pPrivateSet->idGeneration != idGenerationOld + 1;
1327 if (fRedoFromStart)
1328 break;
1329 uLockNanoTS = RTTimeSystemNanoTS();
1330 iCountDown = 10240;
1331 }
1332 }
1333
1334 /* Advance and maybe yield the lock. */
1335 pChunk = pNext;
1336 if (--iCountDown == 0)
1337 {
1338 uint64_t const idGenerationOld = pPrivateSet->idGeneration;
1339 fRedoFromStart = gmmR0MutexYield(pGMM, &uLockNanoTS)
1340 && pPrivateSet->idGeneration != idGenerationOld;
1341 if (fRedoFromStart)
1342 break;
1343 iCountDown = 10240;
1344 }
1345 }
1346 } while (fRedoFromStart);
1347
1348 /*
1349 * Account for shared pages that weren't freed.
1350 */
1351 if (pGVM->gmm.s.Stats.cSharedPages)
1352 {
1353 Assert(pGMM->cSharedPages >= pGVM->gmm.s.Stats.cSharedPages);
1354 SUPR0Printf("GMMR0CleanupVM: hGVM=%#x left %#x shared pages behind!\n", pGVM->hSelf, pGVM->gmm.s.Stats.cSharedPages);
1355 pGMM->cLeftBehindSharedPages += pGVM->gmm.s.Stats.cSharedPages;
1356 }
1357
1358 /*
1359 * Clean up balloon statistics in case the VM process crashed.
1360 */
1361 Assert(pGMM->cBalloonedPages >= pGVM->gmm.s.Stats.cBalloonedPages);
1362 pGMM->cBalloonedPages -= pGVM->gmm.s.Stats.cBalloonedPages;
1363
1364 /*
1365 * Update the over-commitment management statistics.
1366 */
1367 pGMM->cReservedPages -= pGVM->gmm.s.Stats.Reserved.cBasePages
1368 + pGVM->gmm.s.Stats.Reserved.cFixedPages
1369 + pGVM->gmm.s.Stats.Reserved.cShadowPages;
1370 switch (pGVM->gmm.s.Stats.enmPolicy)
1371 {
1372 case GMMOCPOLICY_NO_OC:
1373 break;
1374 default:
1375 /** @todo Update GMM->cOverCommittedPages */
1376 break;
1377 }
1378 }
1379
1380 /* zap the GVM data. */
1381 pGVM->gmm.s.Stats.enmPolicy = GMMOCPOLICY_INVALID;
1382 pGVM->gmm.s.Stats.enmPriority = GMMPRIORITY_INVALID;
1383 pGVM->gmm.s.Stats.fMayAllocate = false;
1384
1385 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
1386 gmmR0MutexRelease(pGMM);
1387
1388 LogFlow(("GMMR0CleanupVM: returns\n"));
1389}
1390
1391
1392/**
1393 * Scan one chunk for private pages belonging to the specified VM.
1394 *
1395 * @note This function may drop the giant mutex!
1396 *
1397 * @returns @c true if we've temporarily dropped the giant mutex, @c false if
1398 * we didn't.
1399 * @param pGMM Pointer to the GMM instance.
1400 * @param pGVM The global VM handle.
1401 * @param pChunk The chunk to scan.
1402 */
1403static bool gmmR0CleanupVMScanChunk(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk)
1404{
1405 Assert(!pGMM->fBoundMemoryMode || pChunk->hGVM == pGVM->hSelf);
1406
1407 /*
1408 * Look for pages belonging to the VM.
1409 * (Perform some internal checks while we're scanning.)
1410 */
1411#ifndef VBOX_STRICT
1412 if (pChunk->cFree != (GMM_CHUNK_SIZE >> PAGE_SHIFT))
1413#endif
1414 {
1415 unsigned cPrivate = 0;
1416 unsigned cShared = 0;
1417 unsigned cFree = 0;
1418
1419 gmmR0UnlinkChunk(pChunk); /* avoiding cFreePages updates. */
1420
1421 uint16_t hGVM = pGVM->hSelf;
1422 unsigned iPage = (GMM_CHUNK_SIZE >> PAGE_SHIFT);
1423 while (iPage-- > 0)
1424 if (GMM_PAGE_IS_PRIVATE(&pChunk->aPages[iPage]))
1425 {
1426 if (pChunk->aPages[iPage].Private.hGVM == hGVM)
1427 {
1428 /*
1429 * Free the page.
1430 *
1431 * The reason for not using gmmR0FreePrivatePage here is that we
1432 * must *not* cause the chunk to be freed from under us - we're in
1433 * an AVL tree walk here.
1434 */
1435 pChunk->aPages[iPage].u = 0;
1436 pChunk->aPages[iPage].Free.iNext = pChunk->iFreeHead;
1437 pChunk->aPages[iPage].Free.u2State = GMM_PAGE_STATE_FREE;
1438 pChunk->iFreeHead = iPage;
1439 pChunk->cPrivate--;
1440 pChunk->cFree++;
1441 pGVM->gmm.s.Stats.cPrivatePages--;
1442 cFree++;
1443 }
1444 else
1445 cPrivate++;
1446 }
1447 else if (GMM_PAGE_IS_FREE(&pChunk->aPages[iPage]))
1448 cFree++;
1449 else
1450 cShared++;
1451
1452 gmmR0SelectSetAndLinkChunk(pGMM, pGVM, pChunk);
1453
1454 /*
1455 * Did it add up?
1456 */
1457 if (RT_UNLIKELY( pChunk->cFree != cFree
1458 || pChunk->cPrivate != cPrivate
1459 || pChunk->cShared != cShared))
1460 {
1461 SUPR0Printf("gmmR0CleanupVMScanChunk: Chunk %p/%#x has bogus stats - free=%d/%d private=%d/%d shared=%d/%d\n",
1462 pChunk, pChunk->Core.Key, pChunk->cFree, cFree, pChunk->cPrivate, cPrivate, pChunk->cShared, cShared);
1463 pChunk->cFree = cFree;
1464 pChunk->cPrivate = cPrivate;
1465 pChunk->cShared = cShared;
1466 }
1467 }
1468
1469 /*
1470 * If not in bound memory mode, we should reset the hGVM field
1471 * if it has our handle in it.
1472 */
1473 if (pChunk->hGVM == pGVM->hSelf)
1474 {
1475 if (!g_pGMM->fBoundMemoryMode)
1476 pChunk->hGVM = NIL_GVM_HANDLE;
1477 else if (pChunk->cFree != GMM_CHUNK_NUM_PAGES)
1478 {
1479 SUPR0Printf("gmmR0CleanupVMScanChunk: %p/%#x: cFree=%#x - it should be 0 in bound mode!\n",
1480 pChunk, pChunk->Core.Key, pChunk->cFree);
1481 AssertMsgFailed(("%p/%#x: cFree=%#x - it should be 0 in bound mode!\n", pChunk, pChunk->Core.Key, pChunk->cFree));
1482
1483 gmmR0UnlinkChunk(pChunk);
1484 pChunk->cFree = GMM_CHUNK_NUM_PAGES;
1485 gmmR0SelectSetAndLinkChunk(pGMM, pGVM, pChunk);
1486 }
1487 }
1488
1489 /*
1490 * Look for a mapping belonging to the terminating VM.
1491 */
1492 GMMR0CHUNKMTXSTATE MtxState;
1493 gmmR0ChunkMutexAcquire(&MtxState, pGMM, pChunk, GMMR0CHUNK_MTX_KEEP_GIANT);
1494 unsigned cMappings = pChunk->cMappingsX;
1495 for (unsigned i = 0; i < cMappings; i++)
1496 if (pChunk->paMappingsX[i].pGVM == pGVM)
1497 {
1498 gmmR0ChunkMutexDropGiant(&MtxState);
1499
1500 RTR0MEMOBJ hMemObj = pChunk->paMappingsX[i].hMapObj;
1501
1502 cMappings--;
1503 if (i < cMappings)
1504 pChunk->paMappingsX[i] = pChunk->paMappingsX[cMappings];
1505 pChunk->paMappingsX[cMappings].pGVM = NULL;
1506 pChunk->paMappingsX[cMappings].hMapObj = NIL_RTR0MEMOBJ;
1507 Assert(pChunk->cMappingsX - 1U == cMappings);
1508 pChunk->cMappingsX = cMappings;
1509
1510 int rc = RTR0MemObjFree(hMemObj, false /* fFreeMappings (NA) */);
1511 if (RT_FAILURE(rc))
1512 {
1513 SUPR0Printf("gmmR0CleanupVMScanChunk: %p/%#x: mapping #%x: RTRMemObjFree(%p,false) -> %d \n",
1514 pChunk, pChunk->Core.Key, i, hMemObj, rc);
1515 AssertRC(rc);
1516 }
1517
1518 gmmR0ChunkMutexRelease(&MtxState, pChunk);
1519 return true;
1520 }
1521
1522 gmmR0ChunkMutexRelease(&MtxState, pChunk);
1523 return false;
1524}
1525
1526
1527/**
1528 * The initial resource reservations.
1529 *
1530 * This will make memory reservations according to policy and priority. If there aren't
1531 * sufficient resources available to sustain the VM this function will fail and all
1532 * future allocations requests will fail as well.
1533 *
1534 * These are just the initial reservations made very very early during the VM creation
1535 * process and will be adjusted later in the GMMR0UpdateReservation call after the
1536 * ring-3 init has completed.
1537 *
1538 * @returns VBox status code.
1539 * @retval VERR_GMM_MEMORY_RESERVATION_DECLINED
1540 * @retval VERR_GMM_
1541 *
1542 * @param pVM The cross context VM structure.
1543 * @param idCpu The VCPU id.
1544 * @param cBasePages The number of pages that may be allocated for the base RAM and ROMs.
1545 * This does not include MMIO2 and similar.
1546 * @param cShadowPages The number of pages that may be allocated for shadow paging structures.
1547 * @param cFixedPages The number of pages that may be allocated for fixed objects like the
1548 * hyper heap, MMIO2 and similar.
1549 * @param enmPolicy The OC policy to use on this VM.
1550 * @param enmPriority The priority in an out-of-memory situation.
1551 *
1552 * @thread The creator thread / EMT.
1553 */
1554GMMR0DECL(int) GMMR0InitialReservation(PVM pVM, VMCPUID idCpu, uint64_t cBasePages, uint32_t cShadowPages, uint32_t cFixedPages,
1555 GMMOCPOLICY enmPolicy, GMMPRIORITY enmPriority)
1556{
1557 LogFlow(("GMMR0InitialReservation: pVM=%p cBasePages=%#llx cShadowPages=%#x cFixedPages=%#x enmPolicy=%d enmPriority=%d\n",
1558 pVM, cBasePages, cShadowPages, cFixedPages, enmPolicy, enmPriority));
1559
1560 /*
1561 * Validate, get basics and take the semaphore.
1562 */
1563 PGMM pGMM;
1564 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
1565 PGVM pGVM;
1566 int rc = GVMMR0ByVMAndEMT(pVM, idCpu, &pGVM);
1567 if (RT_FAILURE(rc))
1568 return rc;
1569
1570 AssertReturn(cBasePages, VERR_INVALID_PARAMETER);
1571 AssertReturn(cShadowPages, VERR_INVALID_PARAMETER);
1572 AssertReturn(cFixedPages, VERR_INVALID_PARAMETER);
1573 AssertReturn(enmPolicy > GMMOCPOLICY_INVALID && enmPolicy < GMMOCPOLICY_END, VERR_INVALID_PARAMETER);
1574 AssertReturn(enmPriority > GMMPRIORITY_INVALID && enmPriority < GMMPRIORITY_END, VERR_INVALID_PARAMETER);
1575
1576 gmmR0MutexAcquire(pGMM);
1577 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
1578 {
1579 if ( !pGVM->gmm.s.Stats.Reserved.cBasePages
1580 && !pGVM->gmm.s.Stats.Reserved.cFixedPages
1581 && !pGVM->gmm.s.Stats.Reserved.cShadowPages)
1582 {
1583 /*
1584 * Check if we can accommodate this.
1585 */
1586 /* ... later ... */
1587 if (RT_SUCCESS(rc))
1588 {
1589 /*
1590 * Update the records.
1591 */
1592 pGVM->gmm.s.Stats.Reserved.cBasePages = cBasePages;
1593 pGVM->gmm.s.Stats.Reserved.cFixedPages = cFixedPages;
1594 pGVM->gmm.s.Stats.Reserved.cShadowPages = cShadowPages;
1595 pGVM->gmm.s.Stats.enmPolicy = enmPolicy;
1596 pGVM->gmm.s.Stats.enmPriority = enmPriority;
1597 pGVM->gmm.s.Stats.fMayAllocate = true;
1598
1599 pGMM->cReservedPages += cBasePages + cFixedPages + cShadowPages;
1600 pGMM->cRegisteredVMs++;
1601 }
1602 }
1603 else
1604 rc = VERR_WRONG_ORDER;
1605 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
1606 }
1607 else
1608 rc = VERR_GMM_IS_NOT_SANE;
1609 gmmR0MutexRelease(pGMM);
1610 LogFlow(("GMMR0InitialReservation: returns %Rrc\n", rc));
1611 return rc;
1612}
1613
1614
1615/**
1616 * VMMR0 request wrapper for GMMR0InitialReservation.
1617 *
1618 * @returns see GMMR0InitialReservation.
1619 * @param pVM The cross context VM structure.
1620 * @param idCpu The VCPU id.
1621 * @param pReq Pointer to the request packet.
1622 */
1623GMMR0DECL(int) GMMR0InitialReservationReq(PVM pVM, VMCPUID idCpu, PGMMINITIALRESERVATIONREQ pReq)
1624{
1625 /*
1626 * Validate input and pass it on.
1627 */
1628 AssertPtrReturn(pVM, VERR_INVALID_POINTER);
1629 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
1630 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(*pReq), ("%#x != %#x\n", pReq->Hdr.cbReq, sizeof(*pReq)), VERR_INVALID_PARAMETER);
1631
1632 return GMMR0InitialReservation(pVM, idCpu, pReq->cBasePages, pReq->cShadowPages, pReq->cFixedPages, pReq->enmPolicy, pReq->enmPriority);
1633}
1634
1635
1636/**
1637 * This updates the memory reservation with the additional MMIO2 and ROM pages.
1638 *
1639 * @returns VBox status code.
1640 * @retval VERR_GMM_MEMORY_RESERVATION_DECLINED
1641 *
1642 * @param pVM The cross context VM structure.
1643 * @param idCpu The VCPU id.
1644 * @param cBasePages The number of pages that may be allocated for the base RAM and ROMs.
1645 * This does not include MMIO2 and similar.
1646 * @param cShadowPages The number of pages that may be allocated for shadow paging structures.
1647 * @param cFixedPages The number of pages that may be allocated for fixed objects like the
1648 * hyper heap, MMIO2 and similar.
1649 *
1650 * @thread EMT.
1651 */
1652GMMR0DECL(int) GMMR0UpdateReservation(PVM pVM, VMCPUID idCpu, uint64_t cBasePages, uint32_t cShadowPages, uint32_t cFixedPages)
1653{
1654 LogFlow(("GMMR0UpdateReservation: pVM=%p cBasePages=%#llx cShadowPages=%#x cFixedPages=%#x\n",
1655 pVM, cBasePages, cShadowPages, cFixedPages));
1656
1657 /*
1658 * Validate, get basics and take the semaphore.
1659 */
1660 PGMM pGMM;
1661 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
1662 PGVM pGVM;
1663 int rc = GVMMR0ByVMAndEMT(pVM, idCpu, &pGVM);
1664 if (RT_FAILURE(rc))
1665 return rc;
1666
1667 AssertReturn(cBasePages, VERR_INVALID_PARAMETER);
1668 AssertReturn(cShadowPages, VERR_INVALID_PARAMETER);
1669 AssertReturn(cFixedPages, VERR_INVALID_PARAMETER);
1670
1671 gmmR0MutexAcquire(pGMM);
1672 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
1673 {
1674 if ( pGVM->gmm.s.Stats.Reserved.cBasePages
1675 && pGVM->gmm.s.Stats.Reserved.cFixedPages
1676 && pGVM->gmm.s.Stats.Reserved.cShadowPages)
1677 {
1678 /*
1679 * Check if we can accommodate this.
1680 */
1681 /* ... later ... */
1682 if (RT_SUCCESS(rc))
1683 {
1684 /*
1685 * Update the records.
1686 */
1687 pGMM->cReservedPages -= pGVM->gmm.s.Stats.Reserved.cBasePages
1688 + pGVM->gmm.s.Stats.Reserved.cFixedPages
1689 + pGVM->gmm.s.Stats.Reserved.cShadowPages;
1690 pGMM->cReservedPages += cBasePages + cFixedPages + cShadowPages;
1691
1692 pGVM->gmm.s.Stats.Reserved.cBasePages = cBasePages;
1693 pGVM->gmm.s.Stats.Reserved.cFixedPages = cFixedPages;
1694 pGVM->gmm.s.Stats.Reserved.cShadowPages = cShadowPages;
1695 }
1696 }
1697 else
1698 rc = VERR_WRONG_ORDER;
1699 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
1700 }
1701 else
1702 rc = VERR_GMM_IS_NOT_SANE;
1703 gmmR0MutexRelease(pGMM);
1704 LogFlow(("GMMR0UpdateReservation: returns %Rrc\n", rc));
1705 return rc;
1706}
1707
1708
1709/**
1710 * VMMR0 request wrapper for GMMR0UpdateReservation.
1711 *
1712 * @returns see GMMR0UpdateReservation.
1713 * @param pVM The cross context VM structure.
1714 * @param idCpu The VCPU id.
1715 * @param pReq Pointer to the request packet.
1716 */
1717GMMR0DECL(int) GMMR0UpdateReservationReq(PVM pVM, VMCPUID idCpu, PGMMUPDATERESERVATIONREQ pReq)
1718{
1719 /*
1720 * Validate input and pass it on.
1721 */
1722 AssertPtrReturn(pVM, VERR_INVALID_POINTER);
1723 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
1724 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(*pReq), ("%#x != %#x\n", pReq->Hdr.cbReq, sizeof(*pReq)), VERR_INVALID_PARAMETER);
1725
1726 return GMMR0UpdateReservation(pVM, idCpu, pReq->cBasePages, pReq->cShadowPages, pReq->cFixedPages);
1727}
1728
1729#ifdef GMMR0_WITH_SANITY_CHECK
1730
1731/**
1732 * Performs sanity checks on a free set.
1733 *
1734 * @returns Error count.
1735 *
1736 * @param pGMM Pointer to the GMM instance.
1737 * @param pSet Pointer to the set.
1738 * @param pszSetName The set name.
1739 * @param pszFunction The function from which it was called.
1740 * @param uLine The line number.
1741 */
1742static uint32_t gmmR0SanityCheckSet(PGMM pGMM, PGMMCHUNKFREESET pSet, const char *pszSetName,
1743 const char *pszFunction, unsigned uLineNo)
1744{
1745 uint32_t cErrors = 0;
1746
1747 /*
1748 * Count the free pages in all the chunks and match it against pSet->cFreePages.
1749 */
1750 uint32_t cPages = 0;
1751 for (unsigned i = 0; i < RT_ELEMENTS(pSet->apLists); i++)
1752 {
1753 for (PGMMCHUNK pCur = pSet->apLists[i]; pCur; pCur = pCur->pFreeNext)
1754 {
1755 /** @todo check that the chunk is hash into the right set. */
1756 cPages += pCur->cFree;
1757 }
1758 }
1759 if (RT_UNLIKELY(cPages != pSet->cFreePages))
1760 {
1761 SUPR0Printf("GMM insanity: found %#x pages in the %s set, expected %#x. (%s, line %u)\n",
1762 cPages, pszSetName, pSet->cFreePages, pszFunction, uLineNo);
1763 cErrors++;
1764 }
1765
1766 return cErrors;
1767}
1768
1769
1770/**
1771 * Performs some sanity checks on the GMM while owning lock.
1772 *
1773 * @returns Error count.
1774 *
1775 * @param pGMM Pointer to the GMM instance.
1776 * @param pszFunction The function from which it is called.
1777 * @param uLineNo The line number.
1778 */
1779static uint32_t gmmR0SanityCheck(PGMM pGMM, const char *pszFunction, unsigned uLineNo)
1780{
1781 uint32_t cErrors = 0;
1782
1783 cErrors += gmmR0SanityCheckSet(pGMM, &pGMM->PrivateX, "private", pszFunction, uLineNo);
1784 cErrors += gmmR0SanityCheckSet(pGMM, &pGMM->Shared, "shared", pszFunction, uLineNo);
1785 /** @todo add more sanity checks. */
1786
1787 return cErrors;
1788}
1789
1790#endif /* GMMR0_WITH_SANITY_CHECK */
1791
1792/**
1793 * Looks up a chunk in the tree and fill in the TLB entry for it.
1794 *
1795 * This is not expected to fail and will bitch if it does.
1796 *
1797 * @returns Pointer to the allocation chunk, NULL if not found.
1798 * @param pGMM Pointer to the GMM instance.
1799 * @param idChunk The ID of the chunk to find.
1800 * @param pTlbe Pointer to the TLB entry.
1801 */
1802static PGMMCHUNK gmmR0GetChunkSlow(PGMM pGMM, uint32_t idChunk, PGMMCHUNKTLBE pTlbe)
1803{
1804 PGMMCHUNK pChunk = (PGMMCHUNK)RTAvlU32Get(&pGMM->pChunks, idChunk);
1805 AssertMsgReturn(pChunk, ("Chunk %#x not found!\n", idChunk), NULL);
1806 pTlbe->idChunk = idChunk;
1807 pTlbe->pChunk = pChunk;
1808 return pChunk;
1809}
1810
1811
1812/**
1813 * Finds a allocation chunk.
1814 *
1815 * This is not expected to fail and will bitch if it does.
1816 *
1817 * @returns Pointer to the allocation chunk, NULL if not found.
1818 * @param pGMM Pointer to the GMM instance.
1819 * @param idChunk The ID of the chunk to find.
1820 */
1821DECLINLINE(PGMMCHUNK) gmmR0GetChunk(PGMM pGMM, uint32_t idChunk)
1822{
1823 /*
1824 * Do a TLB lookup, branch if not in the TLB.
1825 */
1826 PGMMCHUNKTLBE pTlbe = &pGMM->ChunkTLB.aEntries[GMM_CHUNKTLB_IDX(idChunk)];
1827 if ( pTlbe->idChunk != idChunk
1828 || !pTlbe->pChunk)
1829 return gmmR0GetChunkSlow(pGMM, idChunk, pTlbe);
1830 return pTlbe->pChunk;
1831}
1832
1833
1834/**
1835 * Finds a page.
1836 *
1837 * This is not expected to fail and will bitch if it does.
1838 *
1839 * @returns Pointer to the page, NULL if not found.
1840 * @param pGMM Pointer to the GMM instance.
1841 * @param idPage The ID of the page to find.
1842 */
1843DECLINLINE(PGMMPAGE) gmmR0GetPage(PGMM pGMM, uint32_t idPage)
1844{
1845 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, idPage >> GMM_CHUNKID_SHIFT);
1846 if (RT_LIKELY(pChunk))
1847 return &pChunk->aPages[idPage & GMM_PAGEID_IDX_MASK];
1848 return NULL;
1849}
1850
1851
1852/**
1853 * Gets the host physical address for a page given by it's ID.
1854 *
1855 * @returns The host physical address or NIL_RTHCPHYS.
1856 * @param pGMM Pointer to the GMM instance.
1857 * @param idPage The ID of the page to find.
1858 */
1859DECLINLINE(RTHCPHYS) gmmR0GetPageHCPhys(PGMM pGMM, uint32_t idPage)
1860{
1861 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, idPage >> GMM_CHUNKID_SHIFT);
1862 if (RT_LIKELY(pChunk))
1863 return RTR0MemObjGetPagePhysAddr(pChunk->hMemObj, idPage & GMM_PAGEID_IDX_MASK);
1864 return NIL_RTHCPHYS;
1865}
1866
1867
1868/**
1869 * Selects the appropriate free list given the number of free pages.
1870 *
1871 * @returns Free list index.
1872 * @param cFree The number of free pages in the chunk.
1873 */
1874DECLINLINE(unsigned) gmmR0SelectFreeSetList(unsigned cFree)
1875{
1876 unsigned iList = cFree >> GMM_CHUNK_FREE_SET_SHIFT;
1877 AssertMsg(iList < RT_SIZEOFMEMB(GMMCHUNKFREESET, apLists) / RT_SIZEOFMEMB(GMMCHUNKFREESET, apLists[0]),
1878 ("%d (%u)\n", iList, cFree));
1879 return iList;
1880}
1881
1882
1883/**
1884 * Unlinks the chunk from the free list it's currently on (if any).
1885 *
1886 * @param pChunk The allocation chunk.
1887 */
1888DECLINLINE(void) gmmR0UnlinkChunk(PGMMCHUNK pChunk)
1889{
1890 PGMMCHUNKFREESET pSet = pChunk->pSet;
1891 if (RT_LIKELY(pSet))
1892 {
1893 pSet->cFreePages -= pChunk->cFree;
1894 pSet->idGeneration++;
1895
1896 PGMMCHUNK pPrev = pChunk->pFreePrev;
1897 PGMMCHUNK pNext = pChunk->pFreeNext;
1898 if (pPrev)
1899 pPrev->pFreeNext = pNext;
1900 else
1901 pSet->apLists[gmmR0SelectFreeSetList(pChunk->cFree)] = pNext;
1902 if (pNext)
1903 pNext->pFreePrev = pPrev;
1904
1905 pChunk->pSet = NULL;
1906 pChunk->pFreeNext = NULL;
1907 pChunk->pFreePrev = NULL;
1908 }
1909 else
1910 {
1911 Assert(!pChunk->pFreeNext);
1912 Assert(!pChunk->pFreePrev);
1913 Assert(!pChunk->cFree);
1914 }
1915}
1916
1917
1918/**
1919 * Links the chunk onto the appropriate free list in the specified free set.
1920 *
1921 * If no free entries, it's not linked into any list.
1922 *
1923 * @param pChunk The allocation chunk.
1924 * @param pSet The free set.
1925 */
1926DECLINLINE(void) gmmR0LinkChunk(PGMMCHUNK pChunk, PGMMCHUNKFREESET pSet)
1927{
1928 Assert(!pChunk->pSet);
1929 Assert(!pChunk->pFreeNext);
1930 Assert(!pChunk->pFreePrev);
1931
1932 if (pChunk->cFree > 0)
1933 {
1934 pChunk->pSet = pSet;
1935 pChunk->pFreePrev = NULL;
1936 unsigned const iList = gmmR0SelectFreeSetList(pChunk->cFree);
1937 pChunk->pFreeNext = pSet->apLists[iList];
1938 if (pChunk->pFreeNext)
1939 pChunk->pFreeNext->pFreePrev = pChunk;
1940 pSet->apLists[iList] = pChunk;
1941
1942 pSet->cFreePages += pChunk->cFree;
1943 pSet->idGeneration++;
1944 }
1945}
1946
1947
1948/**
1949 * Links the chunk onto the appropriate free list in the specified free set.
1950 *
1951 * If no free entries, it's not linked into any list.
1952 *
1953 * @param pGMM Pointer to the GMM instance.
1954 * @param pGVM Pointer to the kernel-only VM instace data.
1955 * @param pChunk The allocation chunk.
1956 */
1957DECLINLINE(void) gmmR0SelectSetAndLinkChunk(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk)
1958{
1959 PGMMCHUNKFREESET pSet;
1960 if (pGMM->fBoundMemoryMode)
1961 pSet = &pGVM->gmm.s.Private;
1962 else if (pChunk->cShared)
1963 pSet = &pGMM->Shared;
1964 else
1965 pSet = &pGMM->PrivateX;
1966 gmmR0LinkChunk(pChunk, pSet);
1967}
1968
1969
1970/**
1971 * Frees a Chunk ID.
1972 *
1973 * @param pGMM Pointer to the GMM instance.
1974 * @param idChunk The Chunk ID to free.
1975 */
1976static void gmmR0FreeChunkId(PGMM pGMM, uint32_t idChunk)
1977{
1978 AssertReturnVoid(idChunk != NIL_GMM_CHUNKID);
1979 AssertMsg(ASMBitTest(&pGMM->bmChunkId[0], idChunk), ("%#x\n", idChunk));
1980 ASMAtomicBitClear(&pGMM->bmChunkId[0], idChunk);
1981}
1982
1983
1984/**
1985 * Allocates a new Chunk ID.
1986 *
1987 * @returns The Chunk ID.
1988 * @param pGMM Pointer to the GMM instance.
1989 */
1990static uint32_t gmmR0AllocateChunkId(PGMM pGMM)
1991{
1992 AssertCompile(!((GMM_CHUNKID_LAST + 1) & 31)); /* must be a multiple of 32 */
1993 AssertCompile(NIL_GMM_CHUNKID == 0);
1994
1995 /*
1996 * Try the next sequential one.
1997 */
1998 int32_t idChunk = ++pGMM->idChunkPrev;
1999#if 0 /** @todo enable this code */
2000 if ( idChunk <= GMM_CHUNKID_LAST
2001 && idChunk > NIL_GMM_CHUNKID
2002 && !ASMAtomicBitTestAndSet(&pVMM->bmChunkId[0], idChunk))
2003 return idChunk;
2004#endif
2005
2006 /*
2007 * Scan sequentially from the last one.
2008 */
2009 if ( (uint32_t)idChunk < GMM_CHUNKID_LAST
2010 && idChunk > NIL_GMM_CHUNKID)
2011 {
2012 idChunk = ASMBitNextClear(&pGMM->bmChunkId[0], GMM_CHUNKID_LAST + 1, idChunk - 1);
2013 if (idChunk > NIL_GMM_CHUNKID)
2014 {
2015 AssertMsgReturn(!ASMAtomicBitTestAndSet(&pGMM->bmChunkId[0], idChunk), ("%#x\n", idChunk), NIL_GMM_CHUNKID);
2016 return pGMM->idChunkPrev = idChunk;
2017 }
2018 }
2019
2020 /*
2021 * Ok, scan from the start.
2022 * We're not racing anyone, so there is no need to expect failures or have restart loops.
2023 */
2024 idChunk = ASMBitFirstClear(&pGMM->bmChunkId[0], GMM_CHUNKID_LAST + 1);
2025 AssertMsgReturn(idChunk > NIL_GMM_CHUNKID, ("%#x\n", idChunk), NIL_GVM_HANDLE);
2026 AssertMsgReturn(!ASMAtomicBitTestAndSet(&pGMM->bmChunkId[0], idChunk), ("%#x\n", idChunk), NIL_GMM_CHUNKID);
2027
2028 return pGMM->idChunkPrev = idChunk;
2029}
2030
2031
2032/**
2033 * Allocates one private page.
2034 *
2035 * Worker for gmmR0AllocatePages.
2036 *
2037 * @param pChunk The chunk to allocate it from.
2038 * @param hGVM The GVM handle of the VM requesting memory.
2039 * @param pPageDesc The page descriptor.
2040 */
2041static void gmmR0AllocatePage(PGMMCHUNK pChunk, uint32_t hGVM, PGMMPAGEDESC pPageDesc)
2042{
2043 /* update the chunk stats. */
2044 if (pChunk->hGVM == NIL_GVM_HANDLE)
2045 pChunk->hGVM = hGVM;
2046 Assert(pChunk->cFree);
2047 pChunk->cFree--;
2048 pChunk->cPrivate++;
2049
2050 /* unlink the first free page. */
2051 const uint32_t iPage = pChunk->iFreeHead;
2052 AssertReleaseMsg(iPage < RT_ELEMENTS(pChunk->aPages), ("%d\n", iPage));
2053 PGMMPAGE pPage = &pChunk->aPages[iPage];
2054 Assert(GMM_PAGE_IS_FREE(pPage));
2055 pChunk->iFreeHead = pPage->Free.iNext;
2056 Log3(("A pPage=%p iPage=%#x/%#x u2State=%d iFreeHead=%#x iNext=%#x\n",
2057 pPage, iPage, (pChunk->Core.Key << GMM_CHUNKID_SHIFT) | iPage,
2058 pPage->Common.u2State, pChunk->iFreeHead, pPage->Free.iNext));
2059
2060 /* make the page private. */
2061 pPage->u = 0;
2062 AssertCompile(GMM_PAGE_STATE_PRIVATE == 0);
2063 pPage->Private.hGVM = hGVM;
2064 AssertCompile(NIL_RTHCPHYS >= GMM_GCPHYS_LAST);
2065 AssertCompile(GMM_GCPHYS_UNSHAREABLE >= GMM_GCPHYS_LAST);
2066 if (pPageDesc->HCPhysGCPhys <= GMM_GCPHYS_LAST)
2067 pPage->Private.pfn = pPageDesc->HCPhysGCPhys >> PAGE_SHIFT;
2068 else
2069 pPage->Private.pfn = GMM_PAGE_PFN_UNSHAREABLE; /* unshareable / unassigned - same thing. */
2070
2071 /* update the page descriptor. */
2072 pPageDesc->HCPhysGCPhys = RTR0MemObjGetPagePhysAddr(pChunk->hMemObj, iPage);
2073 Assert(pPageDesc->HCPhysGCPhys != NIL_RTHCPHYS);
2074 pPageDesc->idPage = (pChunk->Core.Key << GMM_CHUNKID_SHIFT) | iPage;
2075 pPageDesc->idSharedPage = NIL_GMM_PAGEID;
2076}
2077
2078
2079/**
2080 * Picks the free pages from a chunk.
2081 *
2082 * @returns The new page descriptor table index.
2083 * @param pChunk The chunk.
2084 * @param hGVM The affinity of the chunk. NIL_GVM_HANDLE for no
2085 * affinity.
2086 * @param iPage The current page descriptor table index.
2087 * @param cPages The total number of pages to allocate.
2088 * @param paPages The page descriptor table (input + ouput).
2089 */
2090static uint32_t gmmR0AllocatePagesFromChunk(PGMMCHUNK pChunk, uint16_t const hGVM, uint32_t iPage, uint32_t cPages,
2091 PGMMPAGEDESC paPages)
2092{
2093 PGMMCHUNKFREESET pSet = pChunk->pSet; Assert(pSet);
2094 gmmR0UnlinkChunk(pChunk);
2095
2096 for (; pChunk->cFree && iPage < cPages; iPage++)
2097 gmmR0AllocatePage(pChunk, hGVM, &paPages[iPage]);
2098
2099 gmmR0LinkChunk(pChunk, pSet);
2100 return iPage;
2101}
2102
2103
2104/**
2105 * Registers a new chunk of memory.
2106 *
2107 * This is called by both gmmR0AllocateOneChunk and GMMR0SeedChunk.
2108 *
2109 * @returns VBox status code. On success, the giant GMM lock will be held, the
2110 * caller must release it (ugly).
2111 * @param pGMM Pointer to the GMM instance.
2112 * @param pSet Pointer to the set.
2113 * @param MemObj The memory object for the chunk.
2114 * @param hGVM The affinity of the chunk. NIL_GVM_HANDLE for no
2115 * affinity.
2116 * @param fChunkFlags The chunk flags, GMM_CHUNK_FLAGS_XXX.
2117 * @param ppChunk Chunk address (out). Optional.
2118 *
2119 * @remarks The caller must not own the giant GMM mutex.
2120 * The giant GMM mutex will be acquired and returned acquired in
2121 * the success path. On failure, no locks will be held.
2122 */
2123static int gmmR0RegisterChunk(PGMM pGMM, PGMMCHUNKFREESET pSet, RTR0MEMOBJ MemObj, uint16_t hGVM, uint16_t fChunkFlags,
2124 PGMMCHUNK *ppChunk)
2125{
2126 Assert(pGMM->hMtxOwner != RTThreadNativeSelf());
2127 Assert(hGVM != NIL_GVM_HANDLE || pGMM->fBoundMemoryMode);
2128 Assert(fChunkFlags == 0 || fChunkFlags == GMM_CHUNK_FLAGS_LARGE_PAGE);
2129
2130 int rc;
2131 PGMMCHUNK pChunk = (PGMMCHUNK)RTMemAllocZ(sizeof(*pChunk));
2132 if (pChunk)
2133 {
2134 /*
2135 * Initialize it.
2136 */
2137 pChunk->hMemObj = MemObj;
2138 pChunk->cFree = GMM_CHUNK_NUM_PAGES;
2139 pChunk->hGVM = hGVM;
2140 /*pChunk->iFreeHead = 0;*/
2141 pChunk->idNumaNode = gmmR0GetCurrentNumaNodeId();
2142 pChunk->iChunkMtx = UINT8_MAX;
2143 pChunk->fFlags = fChunkFlags;
2144 for (unsigned iPage = 0; iPage < RT_ELEMENTS(pChunk->aPages) - 1; iPage++)
2145 {
2146 pChunk->aPages[iPage].Free.u2State = GMM_PAGE_STATE_FREE;
2147 pChunk->aPages[iPage].Free.iNext = iPage + 1;
2148 }
2149 pChunk->aPages[RT_ELEMENTS(pChunk->aPages) - 1].Free.u2State = GMM_PAGE_STATE_FREE;
2150 pChunk->aPages[RT_ELEMENTS(pChunk->aPages) - 1].Free.iNext = UINT16_MAX;
2151
2152 /*
2153 * Allocate a Chunk ID and insert it into the tree.
2154 * This has to be done behind the mutex of course.
2155 */
2156 rc = gmmR0MutexAcquire(pGMM);
2157 if (RT_SUCCESS(rc))
2158 {
2159 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
2160 {
2161 pChunk->Core.Key = gmmR0AllocateChunkId(pGMM);
2162 if ( pChunk->Core.Key != NIL_GMM_CHUNKID
2163 && pChunk->Core.Key <= GMM_CHUNKID_LAST
2164 && RTAvlU32Insert(&pGMM->pChunks, &pChunk->Core))
2165 {
2166 pGMM->cChunks++;
2167 RTListAppend(&pGMM->ChunkList, &pChunk->ListNode);
2168 gmmR0LinkChunk(pChunk, pSet);
2169 LogFlow(("gmmR0RegisterChunk: pChunk=%p id=%#x cChunks=%d\n", pChunk, pChunk->Core.Key, pGMM->cChunks));
2170
2171 if (ppChunk)
2172 *ppChunk = pChunk;
2173 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
2174 return VINF_SUCCESS;
2175 }
2176
2177 /* bail out */
2178 rc = VERR_GMM_CHUNK_INSERT;
2179 }
2180 else
2181 rc = VERR_GMM_IS_NOT_SANE;
2182 gmmR0MutexRelease(pGMM);
2183 }
2184
2185 RTMemFree(pChunk);
2186 }
2187 else
2188 rc = VERR_NO_MEMORY;
2189 return rc;
2190}
2191
2192
2193/**
2194 * Allocate a new chunk, immediately pick the requested pages from it, and adds
2195 * what's remaining to the specified free set.
2196 *
2197 * @note This will leave the giant mutex while allocating the new chunk!
2198 *
2199 * @returns VBox status code.
2200 * @param pGMM Pointer to the GMM instance data.
2201 * @param pGVM Pointer to the kernel-only VM instace data.
2202 * @param pSet Pointer to the free set.
2203 * @param cPages The number of pages requested.
2204 * @param paPages The page descriptor table (input + output).
2205 * @param piPage The pointer to the page descriptor table index variable.
2206 * This will be updated.
2207 */
2208static int gmmR0AllocateChunkNew(PGMM pGMM, PGVM pGVM, PGMMCHUNKFREESET pSet, uint32_t cPages,
2209 PGMMPAGEDESC paPages, uint32_t *piPage)
2210{
2211 gmmR0MutexRelease(pGMM);
2212
2213 RTR0MEMOBJ hMemObj;
2214 int rc = RTR0MemObjAllocPhysNC(&hMemObj, GMM_CHUNK_SIZE, NIL_RTHCPHYS);
2215 if (RT_SUCCESS(rc))
2216 {
2217/** @todo Duplicate gmmR0RegisterChunk here so we can avoid chaining up the
2218 * free pages first and then unchaining them right afterwards. Instead
2219 * do as much work as possible without holding the giant lock. */
2220 PGMMCHUNK pChunk;
2221 rc = gmmR0RegisterChunk(pGMM, pSet, hMemObj, pGVM->hSelf, 0 /*fChunkFlags*/, &pChunk);
2222 if (RT_SUCCESS(rc))
2223 {
2224 *piPage = gmmR0AllocatePagesFromChunk(pChunk, pGVM->hSelf, *piPage, cPages, paPages);
2225 return VINF_SUCCESS;
2226 }
2227
2228 /* bail out */
2229 RTR0MemObjFree(hMemObj, false /* fFreeMappings */);
2230 }
2231
2232 int rc2 = gmmR0MutexAcquire(pGMM);
2233 AssertRCReturn(rc2, RT_FAILURE(rc) ? rc : rc2);
2234 return rc;
2235
2236}
2237
2238
2239/**
2240 * As a last restort we'll pick any page we can get.
2241 *
2242 * @returns The new page descriptor table index.
2243 * @param pSet The set to pick from.
2244 * @param pGVM Pointer to the global VM structure.
2245 * @param iPage The current page descriptor table index.
2246 * @param cPages The total number of pages to allocate.
2247 * @param paPages The page descriptor table (input + ouput).
2248 */
2249static uint32_t gmmR0AllocatePagesIndiscriminately(PGMMCHUNKFREESET pSet, PGVM pGVM,
2250 uint32_t iPage, uint32_t cPages, PGMMPAGEDESC paPages)
2251{
2252 unsigned iList = RT_ELEMENTS(pSet->apLists);
2253 while (iList-- > 0)
2254 {
2255 PGMMCHUNK pChunk = pSet->apLists[iList];
2256 while (pChunk)
2257 {
2258 PGMMCHUNK pNext = pChunk->pFreeNext;
2259
2260 iPage = gmmR0AllocatePagesFromChunk(pChunk, pGVM->hSelf, iPage, cPages, paPages);
2261 if (iPage >= cPages)
2262 return iPage;
2263
2264 pChunk = pNext;
2265 }
2266 }
2267 return iPage;
2268}
2269
2270
2271/**
2272 * Pick pages from empty chunks on the same NUMA node.
2273 *
2274 * @returns The new page descriptor table index.
2275 * @param pSet The set to pick from.
2276 * @param pGVM Pointer to the global VM structure.
2277 * @param iPage The current page descriptor table index.
2278 * @param cPages The total number of pages to allocate.
2279 * @param paPages The page descriptor table (input + ouput).
2280 */
2281static uint32_t gmmR0AllocatePagesFromEmptyChunksOnSameNode(PGMMCHUNKFREESET pSet, PGVM pGVM,
2282 uint32_t iPage, uint32_t cPages, PGMMPAGEDESC paPages)
2283{
2284 PGMMCHUNK pChunk = pSet->apLists[GMM_CHUNK_FREE_SET_UNUSED_LIST];
2285 if (pChunk)
2286 {
2287 uint16_t const idNumaNode = gmmR0GetCurrentNumaNodeId();
2288 while (pChunk)
2289 {
2290 PGMMCHUNK pNext = pChunk->pFreeNext;
2291
2292 if (pChunk->idNumaNode == idNumaNode)
2293 {
2294 pChunk->hGVM = pGVM->hSelf;
2295 iPage = gmmR0AllocatePagesFromChunk(pChunk, pGVM->hSelf, iPage, cPages, paPages);
2296 if (iPage >= cPages)
2297 {
2298 pGVM->gmm.s.idLastChunkHint = pChunk->cFree ? pChunk->Core.Key : NIL_GMM_CHUNKID;
2299 return iPage;
2300 }
2301 }
2302
2303 pChunk = pNext;
2304 }
2305 }
2306 return iPage;
2307}
2308
2309
2310/**
2311 * Pick pages from non-empty chunks on the same NUMA node.
2312 *
2313 * @returns The new page descriptor table index.
2314 * @param pSet The set to pick from.
2315 * @param pGVM Pointer to the global VM structure.
2316 * @param iPage The current page descriptor table index.
2317 * @param cPages The total number of pages to allocate.
2318 * @param paPages The page descriptor table (input + ouput).
2319 */
2320static uint32_t gmmR0AllocatePagesFromSameNode(PGMMCHUNKFREESET pSet, PGVM pGVM,
2321 uint32_t iPage, uint32_t cPages, PGMMPAGEDESC paPages)
2322{
2323 /** @todo start by picking from chunks with about the right size first? */
2324 uint16_t const idNumaNode = gmmR0GetCurrentNumaNodeId();
2325 unsigned iList = GMM_CHUNK_FREE_SET_UNUSED_LIST;
2326 while (iList-- > 0)
2327 {
2328 PGMMCHUNK pChunk = pSet->apLists[iList];
2329 while (pChunk)
2330 {
2331 PGMMCHUNK pNext = pChunk->pFreeNext;
2332
2333 if (pChunk->idNumaNode == idNumaNode)
2334 {
2335 iPage = gmmR0AllocatePagesFromChunk(pChunk, pGVM->hSelf, iPage, cPages, paPages);
2336 if (iPage >= cPages)
2337 {
2338 pGVM->gmm.s.idLastChunkHint = pChunk->cFree ? pChunk->Core.Key : NIL_GMM_CHUNKID;
2339 return iPage;
2340 }
2341 }
2342
2343 pChunk = pNext;
2344 }
2345 }
2346 return iPage;
2347}
2348
2349
2350/**
2351 * Pick pages that are in chunks already associated with the VM.
2352 *
2353 * @returns The new page descriptor table index.
2354 * @param pGMM Pointer to the GMM instance data.
2355 * @param pGVM Pointer to the global VM structure.
2356 * @param pSet The set to pick from.
2357 * @param iPage The current page descriptor table index.
2358 * @param cPages The total number of pages to allocate.
2359 * @param paPages The page descriptor table (input + ouput).
2360 */
2361static uint32_t gmmR0AllocatePagesAssociatedWithVM(PGMM pGMM, PGVM pGVM, PGMMCHUNKFREESET pSet,
2362 uint32_t iPage, uint32_t cPages, PGMMPAGEDESC paPages)
2363{
2364 uint16_t const hGVM = pGVM->hSelf;
2365
2366 /* Hint. */
2367 if (pGVM->gmm.s.idLastChunkHint != NIL_GMM_CHUNKID)
2368 {
2369 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, pGVM->gmm.s.idLastChunkHint);
2370 if (pChunk && pChunk->cFree)
2371 {
2372 iPage = gmmR0AllocatePagesFromChunk(pChunk, hGVM, iPage, cPages, paPages);
2373 if (iPage >= cPages)
2374 return iPage;
2375 }
2376 }
2377
2378 /* Scan. */
2379 for (unsigned iList = 0; iList < RT_ELEMENTS(pSet->apLists); iList++)
2380 {
2381 PGMMCHUNK pChunk = pSet->apLists[iList];
2382 while (pChunk)
2383 {
2384 PGMMCHUNK pNext = pChunk->pFreeNext;
2385
2386 if (pChunk->hGVM == hGVM)
2387 {
2388 iPage = gmmR0AllocatePagesFromChunk(pChunk, hGVM, iPage, cPages, paPages);
2389 if (iPage >= cPages)
2390 {
2391 pGVM->gmm.s.idLastChunkHint = pChunk->cFree ? pChunk->Core.Key : NIL_GMM_CHUNKID;
2392 return iPage;
2393 }
2394 }
2395
2396 pChunk = pNext;
2397 }
2398 }
2399 return iPage;
2400}
2401
2402
2403
2404/**
2405 * Pick pages in bound memory mode.
2406 *
2407 * @returns The new page descriptor table index.
2408 * @param pGVM Pointer to the global VM structure.
2409 * @param iPage The current page descriptor table index.
2410 * @param cPages The total number of pages to allocate.
2411 * @param paPages The page descriptor table (input + ouput).
2412 */
2413static uint32_t gmmR0AllocatePagesInBoundMode(PGVM pGVM, uint32_t iPage, uint32_t cPages, PGMMPAGEDESC paPages)
2414{
2415 for (unsigned iList = 0; iList < RT_ELEMENTS(pGVM->gmm.s.Private.apLists); iList++)
2416 {
2417 PGMMCHUNK pChunk = pGVM->gmm.s.Private.apLists[iList];
2418 while (pChunk)
2419 {
2420 Assert(pChunk->hGVM == pGVM->hSelf);
2421 PGMMCHUNK pNext = pChunk->pFreeNext;
2422 iPage = gmmR0AllocatePagesFromChunk(pChunk, pGVM->hSelf, iPage, cPages, paPages);
2423 if (iPage >= cPages)
2424 return iPage;
2425 pChunk = pNext;
2426 }
2427 }
2428 return iPage;
2429}
2430
2431
2432/**
2433 * Checks if we should start picking pages from chunks of other VMs because
2434 * we're getting close to the system memory or reserved limit.
2435 *
2436 * @returns @c true if we should, @c false if we should first try allocate more
2437 * chunks.
2438 */
2439static bool gmmR0ShouldAllocatePagesInOtherChunksBecauseOfLimits(PGVM pGVM)
2440{
2441 /*
2442 * Don't allocate a new chunk if we're
2443 */
2444 uint64_t cPgReserved = pGVM->gmm.s.Stats.Reserved.cBasePages
2445 + pGVM->gmm.s.Stats.Reserved.cFixedPages
2446 - pGVM->gmm.s.Stats.cBalloonedPages
2447 /** @todo what about shared pages? */;
2448 uint64_t cPgAllocated = pGVM->gmm.s.Stats.Allocated.cBasePages
2449 + pGVM->gmm.s.Stats.Allocated.cFixedPages;
2450 uint64_t cPgDelta = cPgReserved - cPgAllocated;
2451 if (cPgDelta < GMM_CHUNK_NUM_PAGES * 4)
2452 return true;
2453 /** @todo make the threshold configurable, also test the code to see if
2454 * this ever kicks in (we might be reserving too much or smth). */
2455
2456 /*
2457 * Check how close we're to the max memory limit and how many fragments
2458 * there are?...
2459 */
2460 /** @todo. */
2461
2462 return false;
2463}
2464
2465
2466/**
2467 * Checks if we should start picking pages from chunks of other VMs because
2468 * there is a lot of free pages around.
2469 *
2470 * @returns @c true if we should, @c false if we should first try allocate more
2471 * chunks.
2472 */
2473static bool gmmR0ShouldAllocatePagesInOtherChunksBecauseOfLotsFree(PGMM pGMM)
2474{
2475 /*
2476 * Setting the limit at 16 chunks (32 MB) at the moment.
2477 */
2478 if (pGMM->PrivateX.cFreePages >= GMM_CHUNK_NUM_PAGES * 16)
2479 return true;
2480 return false;
2481}
2482
2483
2484/**
2485 * Common worker for GMMR0AllocateHandyPages and GMMR0AllocatePages.
2486 *
2487 * @returns VBox status code:
2488 * @retval VINF_SUCCESS on success.
2489 * @retval VERR_GMM_SEED_ME if seeding via GMMR0SeedChunk or
2490 * gmmR0AllocateMoreChunks is necessary.
2491 * @retval VERR_GMM_HIT_GLOBAL_LIMIT if we've exhausted the available pages.
2492 * @retval VERR_GMM_HIT_VM_ACCOUNT_LIMIT if we've hit the VM account limit,
2493 * that is we're trying to allocate more than we've reserved.
2494 *
2495 * @param pGMM Pointer to the GMM instance data.
2496 * @param pGVM Pointer to the VM.
2497 * @param cPages The number of pages to allocate.
2498 * @param paPages Pointer to the page descriptors. See GMMPAGEDESC for
2499 * details on what is expected on input.
2500 * @param enmAccount The account to charge.
2501 *
2502 * @remarks Call takes the giant GMM lock.
2503 */
2504static int gmmR0AllocatePagesNew(PGMM pGMM, PGVM pGVM, uint32_t cPages, PGMMPAGEDESC paPages, GMMACCOUNT enmAccount)
2505{
2506 Assert(pGMM->hMtxOwner == RTThreadNativeSelf());
2507
2508 /*
2509 * Check allocation limits.
2510 */
2511 if (RT_UNLIKELY(pGMM->cAllocatedPages + cPages > pGMM->cMaxPages))
2512 return VERR_GMM_HIT_GLOBAL_LIMIT;
2513
2514 switch (enmAccount)
2515 {
2516 case GMMACCOUNT_BASE:
2517 if (RT_UNLIKELY( pGVM->gmm.s.Stats.Allocated.cBasePages + pGVM->gmm.s.Stats.cBalloonedPages + cPages
2518 > pGVM->gmm.s.Stats.Reserved.cBasePages))
2519 {
2520 Log(("gmmR0AllocatePages:Base: Reserved=%#llx Allocated+Ballooned+Requested=%#llx+%#llx+%#x!\n",
2521 pGVM->gmm.s.Stats.Reserved.cBasePages, pGVM->gmm.s.Stats.Allocated.cBasePages,
2522 pGVM->gmm.s.Stats.cBalloonedPages, cPages));
2523 return VERR_GMM_HIT_VM_ACCOUNT_LIMIT;
2524 }
2525 break;
2526 case GMMACCOUNT_SHADOW:
2527 if (RT_UNLIKELY(pGVM->gmm.s.Stats.Allocated.cShadowPages + cPages > pGVM->gmm.s.Stats.Reserved.cShadowPages))
2528 {
2529 Log(("gmmR0AllocatePages:Shadow: Reserved=%#x Allocated+Requested=%#x+%#x!\n",
2530 pGVM->gmm.s.Stats.Reserved.cShadowPages, pGVM->gmm.s.Stats.Allocated.cShadowPages, cPages));
2531 return VERR_GMM_HIT_VM_ACCOUNT_LIMIT;
2532 }
2533 break;
2534 case GMMACCOUNT_FIXED:
2535 if (RT_UNLIKELY(pGVM->gmm.s.Stats.Allocated.cFixedPages + cPages > pGVM->gmm.s.Stats.Reserved.cFixedPages))
2536 {
2537 Log(("gmmR0AllocatePages:Fixed: Reserved=%#x Allocated+Requested=%#x+%#x!\n",
2538 pGVM->gmm.s.Stats.Reserved.cFixedPages, pGVM->gmm.s.Stats.Allocated.cFixedPages, cPages));
2539 return VERR_GMM_HIT_VM_ACCOUNT_LIMIT;
2540 }
2541 break;
2542 default:
2543 AssertMsgFailedReturn(("enmAccount=%d\n", enmAccount), VERR_IPE_NOT_REACHED_DEFAULT_CASE);
2544 }
2545
2546 /*
2547 * If we're in legacy memory mode, it's easy to figure if we have
2548 * sufficient number of pages up-front.
2549 */
2550 if ( pGMM->fLegacyAllocationMode
2551 && pGVM->gmm.s.Private.cFreePages < cPages)
2552 {
2553 Assert(pGMM->fBoundMemoryMode);
2554 return VERR_GMM_SEED_ME;
2555 }
2556
2557 /*
2558 * Update the accounts before we proceed because we might be leaving the
2559 * protection of the global mutex and thus run the risk of permitting
2560 * too much memory to be allocated.
2561 */
2562 switch (enmAccount)
2563 {
2564 case GMMACCOUNT_BASE: pGVM->gmm.s.Stats.Allocated.cBasePages += cPages; break;
2565 case GMMACCOUNT_SHADOW: pGVM->gmm.s.Stats.Allocated.cShadowPages += cPages; break;
2566 case GMMACCOUNT_FIXED: pGVM->gmm.s.Stats.Allocated.cFixedPages += cPages; break;
2567 default: AssertMsgFailedReturn(("enmAccount=%d\n", enmAccount), VERR_IPE_NOT_REACHED_DEFAULT_CASE);
2568 }
2569 pGVM->gmm.s.Stats.cPrivatePages += cPages;
2570 pGMM->cAllocatedPages += cPages;
2571
2572 /*
2573 * Part two of it's-easy-in-legacy-memory-mode.
2574 */
2575 uint32_t iPage = 0;
2576 if (pGMM->fLegacyAllocationMode)
2577 {
2578 iPage = gmmR0AllocatePagesInBoundMode(pGVM, iPage, cPages, paPages);
2579 AssertReleaseReturn(iPage == cPages, VERR_GMM_ALLOC_PAGES_IPE);
2580 return VINF_SUCCESS;
2581 }
2582
2583 /*
2584 * Bound mode is also relatively straightforward.
2585 */
2586 int rc = VINF_SUCCESS;
2587 if (pGMM->fBoundMemoryMode)
2588 {
2589 iPage = gmmR0AllocatePagesInBoundMode(pGVM, iPage, cPages, paPages);
2590 if (iPage < cPages)
2591 do
2592 rc = gmmR0AllocateChunkNew(pGMM, pGVM, &pGVM->gmm.s.Private, cPages, paPages, &iPage);
2593 while (iPage < cPages && RT_SUCCESS(rc));
2594 }
2595 /*
2596 * Shared mode is trickier as we should try archive the same locality as
2597 * in bound mode, but smartly make use of non-full chunks allocated by
2598 * other VMs if we're low on memory.
2599 */
2600 else
2601 {
2602 /* Pick the most optimal pages first. */
2603 iPage = gmmR0AllocatePagesAssociatedWithVM(pGMM, pGVM, &pGMM->PrivateX, iPage, cPages, paPages);
2604 if (iPage < cPages)
2605 {
2606 /* Maybe we should try getting pages from chunks "belonging" to
2607 other VMs before allocating more chunks? */
2608 bool fTriedOnSameAlready = false;
2609 if (gmmR0ShouldAllocatePagesInOtherChunksBecauseOfLimits(pGVM))
2610 {
2611 iPage = gmmR0AllocatePagesFromSameNode(&pGMM->PrivateX, pGVM, iPage, cPages, paPages);
2612 fTriedOnSameAlready = true;
2613 }
2614
2615 /* Allocate memory from empty chunks. */
2616 if (iPage < cPages)
2617 iPage = gmmR0AllocatePagesFromEmptyChunksOnSameNode(&pGMM->PrivateX, pGVM, iPage, cPages, paPages);
2618
2619 /* Grab empty shared chunks. */
2620 if (iPage < cPages)
2621 iPage = gmmR0AllocatePagesFromEmptyChunksOnSameNode(&pGMM->Shared, pGVM, iPage, cPages, paPages);
2622
2623 /* If there is a lof of free pages spread around, try not waste
2624 system memory on more chunks. (Should trigger defragmentation.) */
2625 if ( !fTriedOnSameAlready
2626 && gmmR0ShouldAllocatePagesInOtherChunksBecauseOfLotsFree(pGMM))
2627 {
2628 iPage = gmmR0AllocatePagesFromSameNode(&pGMM->PrivateX, pGVM, iPage, cPages, paPages);
2629 if (iPage < cPages)
2630 iPage = gmmR0AllocatePagesIndiscriminately(&pGMM->PrivateX, pGVM, iPage, cPages, paPages);
2631 }
2632
2633 /*
2634 * Ok, try allocate new chunks.
2635 */
2636 if (iPage < cPages)
2637 {
2638 do
2639 rc = gmmR0AllocateChunkNew(pGMM, pGVM, &pGMM->PrivateX, cPages, paPages, &iPage);
2640 while (iPage < cPages && RT_SUCCESS(rc));
2641
2642 /* If the host is out of memory, take whatever we can get. */
2643 if ( (rc == VERR_NO_MEMORY || rc == VERR_NO_PHYS_MEMORY)
2644 && pGMM->PrivateX.cFreePages + pGMM->Shared.cFreePages >= cPages - iPage)
2645 {
2646 iPage = gmmR0AllocatePagesIndiscriminately(&pGMM->PrivateX, pGVM, iPage, cPages, paPages);
2647 if (iPage < cPages)
2648 iPage = gmmR0AllocatePagesIndiscriminately(&pGMM->Shared, pGVM, iPage, cPages, paPages);
2649 AssertRelease(iPage == cPages);
2650 rc = VINF_SUCCESS;
2651 }
2652 }
2653 }
2654 }
2655
2656 /*
2657 * Clean up on failure. Since this is bound to be a low-memory condition
2658 * we will give back any empty chunks that might be hanging around.
2659 */
2660 if (RT_FAILURE(rc))
2661 {
2662 /* Update the statistics. */
2663 pGVM->gmm.s.Stats.cPrivatePages -= cPages;
2664 pGMM->cAllocatedPages -= cPages - iPage;
2665 switch (enmAccount)
2666 {
2667 case GMMACCOUNT_BASE: pGVM->gmm.s.Stats.Allocated.cBasePages -= cPages; break;
2668 case GMMACCOUNT_SHADOW: pGVM->gmm.s.Stats.Allocated.cShadowPages -= cPages; break;
2669 case GMMACCOUNT_FIXED: pGVM->gmm.s.Stats.Allocated.cFixedPages -= cPages; break;
2670 default: AssertMsgFailedReturn(("enmAccount=%d\n", enmAccount), VERR_IPE_NOT_REACHED_DEFAULT_CASE);
2671 }
2672
2673 /* Release the pages. */
2674 while (iPage-- > 0)
2675 {
2676 uint32_t idPage = paPages[iPage].idPage;
2677 PGMMPAGE pPage = gmmR0GetPage(pGMM, idPage);
2678 if (RT_LIKELY(pPage))
2679 {
2680 Assert(GMM_PAGE_IS_PRIVATE(pPage));
2681 Assert(pPage->Private.hGVM == pGVM->hSelf);
2682 gmmR0FreePrivatePage(pGMM, pGVM, idPage, pPage);
2683 }
2684 else
2685 AssertMsgFailed(("idPage=%#x\n", idPage));
2686
2687 paPages[iPage].idPage = NIL_GMM_PAGEID;
2688 paPages[iPage].idSharedPage = NIL_GMM_PAGEID;
2689 paPages[iPage].HCPhysGCPhys = NIL_RTHCPHYS;
2690 }
2691
2692 /* Free empty chunks. */
2693 /** @todo */
2694
2695 /* return the fail status on failure */
2696 return rc;
2697 }
2698 return VINF_SUCCESS;
2699}
2700
2701
2702/**
2703 * Updates the previous allocations and allocates more pages.
2704 *
2705 * The handy pages are always taken from the 'base' memory account.
2706 * The allocated pages are not cleared and will contains random garbage.
2707 *
2708 * @returns VBox status code:
2709 * @retval VINF_SUCCESS on success.
2710 * @retval VERR_NOT_OWNER if the caller is not an EMT.
2711 * @retval VERR_GMM_PAGE_NOT_FOUND if one of the pages to update wasn't found.
2712 * @retval VERR_GMM_PAGE_NOT_PRIVATE if one of the pages to update wasn't a
2713 * private page.
2714 * @retval VERR_GMM_PAGE_NOT_SHARED if one of the pages to update wasn't a
2715 * shared page.
2716 * @retval VERR_GMM_NOT_PAGE_OWNER if one of the pages to be updated wasn't
2717 * owned by the VM.
2718 * @retval VERR_GMM_SEED_ME if seeding via GMMR0SeedChunk is necessary.
2719 * @retval VERR_GMM_HIT_GLOBAL_LIMIT if we've exhausted the available pages.
2720 * @retval VERR_GMM_HIT_VM_ACCOUNT_LIMIT if we've hit the VM account limit,
2721 * that is we're trying to allocate more than we've reserved.
2722 *
2723 * @param pVM The cross context VM structure.
2724 * @param idCpu The VCPU id.
2725 * @param cPagesToUpdate The number of pages to update (starting from the head).
2726 * @param cPagesToAlloc The number of pages to allocate (starting from the head).
2727 * @param paPages The array of page descriptors.
2728 * See GMMPAGEDESC for details on what is expected on input.
2729 * @thread EMT.
2730 */
2731GMMR0DECL(int) GMMR0AllocateHandyPages(PVM pVM, VMCPUID idCpu, uint32_t cPagesToUpdate, uint32_t cPagesToAlloc, PGMMPAGEDESC paPages)
2732{
2733 LogFlow(("GMMR0AllocateHandyPages: pVM=%p cPagesToUpdate=%#x cPagesToAlloc=%#x paPages=%p\n",
2734 pVM, cPagesToUpdate, cPagesToAlloc, paPages));
2735
2736 /*
2737 * Validate, get basics and take the semaphore.
2738 * (This is a relatively busy path, so make predictions where possible.)
2739 */
2740 PGMM pGMM;
2741 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
2742 PGVM pGVM;
2743 int rc = GVMMR0ByVMAndEMT(pVM, idCpu, &pGVM);
2744 if (RT_FAILURE(rc))
2745 return rc;
2746
2747 AssertPtrReturn(paPages, VERR_INVALID_PARAMETER);
2748 AssertMsgReturn( (cPagesToUpdate && cPagesToUpdate < 1024)
2749 || (cPagesToAlloc && cPagesToAlloc < 1024),
2750 ("cPagesToUpdate=%#x cPagesToAlloc=%#x\n", cPagesToUpdate, cPagesToAlloc),
2751 VERR_INVALID_PARAMETER);
2752
2753 unsigned iPage = 0;
2754 for (; iPage < cPagesToUpdate; iPage++)
2755 {
2756 AssertMsgReturn( ( paPages[iPage].HCPhysGCPhys <= GMM_GCPHYS_LAST
2757 && !(paPages[iPage].HCPhysGCPhys & PAGE_OFFSET_MASK))
2758 || paPages[iPage].HCPhysGCPhys == NIL_RTHCPHYS
2759 || paPages[iPage].HCPhysGCPhys == GMM_GCPHYS_UNSHAREABLE,
2760 ("#%#x: %RHp\n", iPage, paPages[iPage].HCPhysGCPhys),
2761 VERR_INVALID_PARAMETER);
2762 AssertMsgReturn( paPages[iPage].idPage <= GMM_PAGEID_LAST
2763 /*|| paPages[iPage].idPage == NIL_GMM_PAGEID*/,
2764 ("#%#x: %#x\n", iPage, paPages[iPage].idPage), VERR_INVALID_PARAMETER);
2765 AssertMsgReturn( paPages[iPage].idPage <= GMM_PAGEID_LAST
2766 /*|| paPages[iPage].idSharedPage == NIL_GMM_PAGEID*/,
2767 ("#%#x: %#x\n", iPage, paPages[iPage].idSharedPage), VERR_INVALID_PARAMETER);
2768 }
2769
2770 for (; iPage < cPagesToAlloc; iPage++)
2771 {
2772 AssertMsgReturn(paPages[iPage].HCPhysGCPhys == NIL_RTHCPHYS, ("#%#x: %RHp\n", iPage, paPages[iPage].HCPhysGCPhys), VERR_INVALID_PARAMETER);
2773 AssertMsgReturn(paPages[iPage].idPage == NIL_GMM_PAGEID, ("#%#x: %#x\n", iPage, paPages[iPage].idPage), VERR_INVALID_PARAMETER);
2774 AssertMsgReturn(paPages[iPage].idSharedPage == NIL_GMM_PAGEID, ("#%#x: %#x\n", iPage, paPages[iPage].idSharedPage), VERR_INVALID_PARAMETER);
2775 }
2776
2777 gmmR0MutexAcquire(pGMM);
2778 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
2779 {
2780 /* No allocations before the initial reservation has been made! */
2781 if (RT_LIKELY( pGVM->gmm.s.Stats.Reserved.cBasePages
2782 && pGVM->gmm.s.Stats.Reserved.cFixedPages
2783 && pGVM->gmm.s.Stats.Reserved.cShadowPages))
2784 {
2785 /*
2786 * Perform the updates.
2787 * Stop on the first error.
2788 */
2789 for (iPage = 0; iPage < cPagesToUpdate; iPage++)
2790 {
2791 if (paPages[iPage].idPage != NIL_GMM_PAGEID)
2792 {
2793 PGMMPAGE pPage = gmmR0GetPage(pGMM, paPages[iPage].idPage);
2794 if (RT_LIKELY(pPage))
2795 {
2796 if (RT_LIKELY(GMM_PAGE_IS_PRIVATE(pPage)))
2797 {
2798 if (RT_LIKELY(pPage->Private.hGVM == pGVM->hSelf))
2799 {
2800 AssertCompile(NIL_RTHCPHYS > GMM_GCPHYS_LAST && GMM_GCPHYS_UNSHAREABLE > GMM_GCPHYS_LAST);
2801 if (RT_LIKELY(paPages[iPage].HCPhysGCPhys <= GMM_GCPHYS_LAST))
2802 pPage->Private.pfn = paPages[iPage].HCPhysGCPhys >> PAGE_SHIFT;
2803 else if (paPages[iPage].HCPhysGCPhys == GMM_GCPHYS_UNSHAREABLE)
2804 pPage->Private.pfn = GMM_PAGE_PFN_UNSHAREABLE;
2805 /* else: NIL_RTHCPHYS nothing */
2806
2807 paPages[iPage].idPage = NIL_GMM_PAGEID;
2808 paPages[iPage].HCPhysGCPhys = NIL_RTHCPHYS;
2809 }
2810 else
2811 {
2812 Log(("GMMR0AllocateHandyPages: #%#x/%#x: Not owner! hGVM=%#x hSelf=%#x\n",
2813 iPage, paPages[iPage].idPage, pPage->Private.hGVM, pGVM->hSelf));
2814 rc = VERR_GMM_NOT_PAGE_OWNER;
2815 break;
2816 }
2817 }
2818 else
2819 {
2820 Log(("GMMR0AllocateHandyPages: #%#x/%#x: Not private! %.*Rhxs (type %d)\n", iPage, paPages[iPage].idPage, sizeof(*pPage), pPage, pPage->Common.u2State));
2821 rc = VERR_GMM_PAGE_NOT_PRIVATE;
2822 break;
2823 }
2824 }
2825 else
2826 {
2827 Log(("GMMR0AllocateHandyPages: #%#x/%#x: Not found! (private)\n", iPage, paPages[iPage].idPage));
2828 rc = VERR_GMM_PAGE_NOT_FOUND;
2829 break;
2830 }
2831 }
2832
2833 if (paPages[iPage].idSharedPage != NIL_GMM_PAGEID)
2834 {
2835 PGMMPAGE pPage = gmmR0GetPage(pGMM, paPages[iPage].idSharedPage);
2836 if (RT_LIKELY(pPage))
2837 {
2838 if (RT_LIKELY(GMM_PAGE_IS_SHARED(pPage)))
2839 {
2840 AssertCompile(NIL_RTHCPHYS > GMM_GCPHYS_LAST && GMM_GCPHYS_UNSHAREABLE > GMM_GCPHYS_LAST);
2841 Assert(pPage->Shared.cRefs);
2842 Assert(pGVM->gmm.s.Stats.cSharedPages);
2843 Assert(pGVM->gmm.s.Stats.Allocated.cBasePages);
2844
2845 Log(("GMMR0AllocateHandyPages: free shared page %x cRefs=%d\n", paPages[iPage].idSharedPage, pPage->Shared.cRefs));
2846 pGVM->gmm.s.Stats.cSharedPages--;
2847 pGVM->gmm.s.Stats.Allocated.cBasePages--;
2848 if (!--pPage->Shared.cRefs)
2849 gmmR0FreeSharedPage(pGMM, pGVM, paPages[iPage].idSharedPage, pPage);
2850 else
2851 {
2852 Assert(pGMM->cDuplicatePages);
2853 pGMM->cDuplicatePages--;
2854 }
2855
2856 paPages[iPage].idSharedPage = NIL_GMM_PAGEID;
2857 }
2858 else
2859 {
2860 Log(("GMMR0AllocateHandyPages: #%#x/%#x: Not shared!\n", iPage, paPages[iPage].idSharedPage));
2861 rc = VERR_GMM_PAGE_NOT_SHARED;
2862 break;
2863 }
2864 }
2865 else
2866 {
2867 Log(("GMMR0AllocateHandyPages: #%#x/%#x: Not found! (shared)\n", iPage, paPages[iPage].idSharedPage));
2868 rc = VERR_GMM_PAGE_NOT_FOUND;
2869 break;
2870 }
2871 }
2872 } /* for each page to update */
2873
2874 if (RT_SUCCESS(rc) && cPagesToAlloc > 0)
2875 {
2876#if defined(VBOX_STRICT) && 0 /** @todo re-test this later. Appeared to be a PGM init bug. */
2877 for (iPage = 0; iPage < cPagesToAlloc; iPage++)
2878 {
2879 Assert(paPages[iPage].HCPhysGCPhys == NIL_RTHCPHYS);
2880 Assert(paPages[iPage].idPage == NIL_GMM_PAGEID);
2881 Assert(paPages[iPage].idSharedPage == NIL_GMM_PAGEID);
2882 }
2883#endif
2884
2885 /*
2886 * Join paths with GMMR0AllocatePages for the allocation.
2887 * Note! gmmR0AllocateMoreChunks may leave the protection of the mutex!
2888 */
2889 rc = gmmR0AllocatePagesNew(pGMM, pGVM, cPagesToAlloc, paPages, GMMACCOUNT_BASE);
2890 }
2891 }
2892 else
2893 rc = VERR_WRONG_ORDER;
2894 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
2895 }
2896 else
2897 rc = VERR_GMM_IS_NOT_SANE;
2898 gmmR0MutexRelease(pGMM);
2899 LogFlow(("GMMR0AllocateHandyPages: returns %Rrc\n", rc));
2900 return rc;
2901}
2902
2903
2904/**
2905 * Allocate one or more pages.
2906 *
2907 * This is typically used for ROMs and MMIO2 (VRAM) during VM creation.
2908 * The allocated pages are not cleared and will contain random garbage.
2909 *
2910 * @returns VBox status code:
2911 * @retval VINF_SUCCESS on success.
2912 * @retval VERR_NOT_OWNER if the caller is not an EMT.
2913 * @retval VERR_GMM_SEED_ME if seeding via GMMR0SeedChunk is necessary.
2914 * @retval VERR_GMM_HIT_GLOBAL_LIMIT if we've exhausted the available pages.
2915 * @retval VERR_GMM_HIT_VM_ACCOUNT_LIMIT if we've hit the VM account limit,
2916 * that is we're trying to allocate more than we've reserved.
2917 *
2918 * @param pVM The cross context VM structure.
2919 * @param idCpu The VCPU id.
2920 * @param cPages The number of pages to allocate.
2921 * @param paPages Pointer to the page descriptors.
2922 * See GMMPAGEDESC for details on what is expected on
2923 * input.
2924 * @param enmAccount The account to charge.
2925 *
2926 * @thread EMT.
2927 */
2928GMMR0DECL(int) GMMR0AllocatePages(PVM pVM, VMCPUID idCpu, uint32_t cPages, PGMMPAGEDESC paPages, GMMACCOUNT enmAccount)
2929{
2930 LogFlow(("GMMR0AllocatePages: pVM=%p cPages=%#x paPages=%p enmAccount=%d\n", pVM, cPages, paPages, enmAccount));
2931
2932 /*
2933 * Validate, get basics and take the semaphore.
2934 */
2935 PGMM pGMM;
2936 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
2937 PGVM pGVM;
2938 int rc = GVMMR0ByVMAndEMT(pVM, idCpu, &pGVM);
2939 if (RT_FAILURE(rc))
2940 return rc;
2941
2942 AssertPtrReturn(paPages, VERR_INVALID_PARAMETER);
2943 AssertMsgReturn(enmAccount > GMMACCOUNT_INVALID && enmAccount < GMMACCOUNT_END, ("%d\n", enmAccount), VERR_INVALID_PARAMETER);
2944 AssertMsgReturn(cPages > 0 && cPages < RT_BIT(32 - PAGE_SHIFT), ("%#x\n", cPages), VERR_INVALID_PARAMETER);
2945
2946 for (unsigned iPage = 0; iPage < cPages; iPage++)
2947 {
2948 AssertMsgReturn( paPages[iPage].HCPhysGCPhys == NIL_RTHCPHYS
2949 || paPages[iPage].HCPhysGCPhys == GMM_GCPHYS_UNSHAREABLE
2950 || ( enmAccount == GMMACCOUNT_BASE
2951 && paPages[iPage].HCPhysGCPhys <= GMM_GCPHYS_LAST
2952 && !(paPages[iPage].HCPhysGCPhys & PAGE_OFFSET_MASK)),
2953 ("#%#x: %RHp enmAccount=%d\n", iPage, paPages[iPage].HCPhysGCPhys, enmAccount),
2954 VERR_INVALID_PARAMETER);
2955 AssertMsgReturn(paPages[iPage].idPage == NIL_GMM_PAGEID, ("#%#x: %#x\n", iPage, paPages[iPage].idPage), VERR_INVALID_PARAMETER);
2956 AssertMsgReturn(paPages[iPage].idSharedPage == NIL_GMM_PAGEID, ("#%#x: %#x\n", iPage, paPages[iPage].idSharedPage), VERR_INVALID_PARAMETER);
2957 }
2958
2959 gmmR0MutexAcquire(pGMM);
2960 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
2961 {
2962
2963 /* No allocations before the initial reservation has been made! */
2964 if (RT_LIKELY( pGVM->gmm.s.Stats.Reserved.cBasePages
2965 && pGVM->gmm.s.Stats.Reserved.cFixedPages
2966 && pGVM->gmm.s.Stats.Reserved.cShadowPages))
2967 rc = gmmR0AllocatePagesNew(pGMM, pGVM, cPages, paPages, enmAccount);
2968 else
2969 rc = VERR_WRONG_ORDER;
2970 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
2971 }
2972 else
2973 rc = VERR_GMM_IS_NOT_SANE;
2974 gmmR0MutexRelease(pGMM);
2975 LogFlow(("GMMR0AllocatePages: returns %Rrc\n", rc));
2976 return rc;
2977}
2978
2979
2980/**
2981 * VMMR0 request wrapper for GMMR0AllocatePages.
2982 *
2983 * @returns see GMMR0AllocatePages.
2984 * @param pVM The cross context VM structure.
2985 * @param idCpu The VCPU id.
2986 * @param pReq Pointer to the request packet.
2987 */
2988GMMR0DECL(int) GMMR0AllocatePagesReq(PVM pVM, VMCPUID idCpu, PGMMALLOCATEPAGESREQ pReq)
2989{
2990 /*
2991 * Validate input and pass it on.
2992 */
2993 AssertPtrReturn(pVM, VERR_INVALID_POINTER);
2994 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
2995 AssertMsgReturn(pReq->Hdr.cbReq >= RT_UOFFSETOF(GMMALLOCATEPAGESREQ, aPages[0]),
2996 ("%#x < %#x\n", pReq->Hdr.cbReq, RT_UOFFSETOF(GMMALLOCATEPAGESREQ, aPages[0])),
2997 VERR_INVALID_PARAMETER);
2998 AssertMsgReturn(pReq->Hdr.cbReq == RT_UOFFSETOF(GMMALLOCATEPAGESREQ, aPages[pReq->cPages]),
2999 ("%#x != %#x\n", pReq->Hdr.cbReq, RT_UOFFSETOF(GMMALLOCATEPAGESREQ, aPages[pReq->cPages])),
3000 VERR_INVALID_PARAMETER);
3001
3002 return GMMR0AllocatePages(pVM, idCpu, pReq->cPages, &pReq->aPages[0], pReq->enmAccount);
3003}
3004
3005
3006/**
3007 * Allocate a large page to represent guest RAM
3008 *
3009 * The allocated pages are not cleared and will contains random garbage.
3010 *
3011 * @returns VBox status code:
3012 * @retval VINF_SUCCESS on success.
3013 * @retval VERR_NOT_OWNER if the caller is not an EMT.
3014 * @retval VERR_GMM_SEED_ME if seeding via GMMR0SeedChunk is necessary.
3015 * @retval VERR_GMM_HIT_GLOBAL_LIMIT if we've exhausted the available pages.
3016 * @retval VERR_GMM_HIT_VM_ACCOUNT_LIMIT if we've hit the VM account limit,
3017 * that is we're trying to allocate more than we've reserved.
3018 * @returns see GMMR0AllocatePages.
3019 *
3020 * @param pVM The cross context VM structure.
3021 * @param idCpu The VCPU id.
3022 * @param cbPage Large page size.
3023 * @param pIdPage Where to return the GMM page ID of the page.
3024 * @param pHCPhys Where to return the host physical address of the page.
3025 */
3026GMMR0DECL(int) GMMR0AllocateLargePage(PVM pVM, VMCPUID idCpu, uint32_t cbPage, uint32_t *pIdPage, RTHCPHYS *pHCPhys)
3027{
3028 LogFlow(("GMMR0AllocateLargePage: pVM=%p cbPage=%x\n", pVM, cbPage));
3029
3030 AssertReturn(cbPage == GMM_CHUNK_SIZE, VERR_INVALID_PARAMETER);
3031 AssertPtrReturn(pIdPage, VERR_INVALID_PARAMETER);
3032 AssertPtrReturn(pHCPhys, VERR_INVALID_PARAMETER);
3033
3034 /*
3035 * Validate, get basics and take the semaphore.
3036 */
3037 PGMM pGMM;
3038 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
3039 PGVM pGVM;
3040 int rc = GVMMR0ByVMAndEMT(pVM, idCpu, &pGVM);
3041 if (RT_FAILURE(rc))
3042 return rc;
3043
3044 /* Not supported in legacy mode where we allocate the memory in ring 3 and lock it in ring 0. */
3045 if (pGMM->fLegacyAllocationMode)
3046 return VERR_NOT_SUPPORTED;
3047
3048 *pHCPhys = NIL_RTHCPHYS;
3049 *pIdPage = NIL_GMM_PAGEID;
3050
3051 gmmR0MutexAcquire(pGMM);
3052 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
3053 {
3054 const unsigned cPages = (GMM_CHUNK_SIZE >> PAGE_SHIFT);
3055 if (RT_UNLIKELY( pGVM->gmm.s.Stats.Allocated.cBasePages + pGVM->gmm.s.Stats.cBalloonedPages + cPages
3056 > pGVM->gmm.s.Stats.Reserved.cBasePages))
3057 {
3058 Log(("GMMR0AllocateLargePage: Reserved=%#llx Allocated+Requested=%#llx+%#x!\n",
3059 pGVM->gmm.s.Stats.Reserved.cBasePages, pGVM->gmm.s.Stats.Allocated.cBasePages, cPages));
3060 gmmR0MutexRelease(pGMM);
3061 return VERR_GMM_HIT_VM_ACCOUNT_LIMIT;
3062 }
3063
3064 /*
3065 * Allocate a new large page chunk.
3066 *
3067 * Note! We leave the giant GMM lock temporarily as the allocation might
3068 * take a long time. gmmR0RegisterChunk will retake it (ugly).
3069 */
3070 AssertCompile(GMM_CHUNK_SIZE == _2M);
3071 gmmR0MutexRelease(pGMM);
3072
3073 RTR0MEMOBJ hMemObj;
3074 rc = RTR0MemObjAllocPhysEx(&hMemObj, GMM_CHUNK_SIZE, NIL_RTHCPHYS, GMM_CHUNK_SIZE);
3075 if (RT_SUCCESS(rc))
3076 {
3077 PGMMCHUNKFREESET pSet = pGMM->fBoundMemoryMode ? &pGVM->gmm.s.Private : &pGMM->PrivateX;
3078 PGMMCHUNK pChunk;
3079 rc = gmmR0RegisterChunk(pGMM, pSet, hMemObj, pGVM->hSelf, GMM_CHUNK_FLAGS_LARGE_PAGE, &pChunk);
3080 if (RT_SUCCESS(rc))
3081 {
3082 /*
3083 * Allocate all the pages in the chunk.
3084 */
3085 /* Unlink the new chunk from the free list. */
3086 gmmR0UnlinkChunk(pChunk);
3087
3088 /** @todo rewrite this to skip the looping. */
3089 /* Allocate all pages. */
3090 GMMPAGEDESC PageDesc;
3091 gmmR0AllocatePage(pChunk, pGVM->hSelf, &PageDesc);
3092
3093 /* Return the first page as we'll use the whole chunk as one big page. */
3094 *pIdPage = PageDesc.idPage;
3095 *pHCPhys = PageDesc.HCPhysGCPhys;
3096
3097 for (unsigned i = 1; i < cPages; i++)
3098 gmmR0AllocatePage(pChunk, pGVM->hSelf, &PageDesc);
3099
3100 /* Update accounting. */
3101 pGVM->gmm.s.Stats.Allocated.cBasePages += cPages;
3102 pGVM->gmm.s.Stats.cPrivatePages += cPages;
3103 pGMM->cAllocatedPages += cPages;
3104
3105 gmmR0LinkChunk(pChunk, pSet);
3106 gmmR0MutexRelease(pGMM);
3107 }
3108 else
3109 RTR0MemObjFree(hMemObj, false /* fFreeMappings */);
3110 }
3111 }
3112 else
3113 {
3114 gmmR0MutexRelease(pGMM);
3115 rc = VERR_GMM_IS_NOT_SANE;
3116 }
3117
3118 LogFlow(("GMMR0AllocateLargePage: returns %Rrc\n", rc));
3119 return rc;
3120}
3121
3122
3123/**
3124 * Free a large page.
3125 *
3126 * @returns VBox status code:
3127 * @param pVM The cross context VM structure.
3128 * @param idCpu The VCPU id.
3129 * @param idPage The large page id.
3130 */
3131GMMR0DECL(int) GMMR0FreeLargePage(PVM pVM, VMCPUID idCpu, uint32_t idPage)
3132{
3133 LogFlow(("GMMR0FreeLargePage: pVM=%p idPage=%x\n", pVM, idPage));
3134
3135 /*
3136 * Validate, get basics and take the semaphore.
3137 */
3138 PGMM pGMM;
3139 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
3140 PGVM pGVM;
3141 int rc = GVMMR0ByVMAndEMT(pVM, idCpu, &pGVM);
3142 if (RT_FAILURE(rc))
3143 return rc;
3144
3145 /* Not supported in legacy mode where we allocate the memory in ring 3 and lock it in ring 0. */
3146 if (pGMM->fLegacyAllocationMode)
3147 return VERR_NOT_SUPPORTED;
3148
3149 gmmR0MutexAcquire(pGMM);
3150 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
3151 {
3152 const unsigned cPages = (GMM_CHUNK_SIZE >> PAGE_SHIFT);
3153
3154 if (RT_UNLIKELY(pGVM->gmm.s.Stats.Allocated.cBasePages < cPages))
3155 {
3156 Log(("GMMR0FreeLargePage: allocated=%#llx cPages=%#x!\n", pGVM->gmm.s.Stats.Allocated.cBasePages, cPages));
3157 gmmR0MutexRelease(pGMM);
3158 return VERR_GMM_ATTEMPT_TO_FREE_TOO_MUCH;
3159 }
3160
3161 PGMMPAGE pPage = gmmR0GetPage(pGMM, idPage);
3162 if (RT_LIKELY( pPage
3163 && GMM_PAGE_IS_PRIVATE(pPage)))
3164 {
3165 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, idPage >> GMM_CHUNKID_SHIFT);
3166 Assert(pChunk);
3167 Assert(pChunk->cFree < GMM_CHUNK_NUM_PAGES);
3168 Assert(pChunk->cPrivate > 0);
3169
3170 /* Release the memory immediately. */
3171 gmmR0FreeChunk(pGMM, NULL, pChunk, false /*fRelaxedSem*/); /** @todo this can be relaxed too! */
3172
3173 /* Update accounting. */
3174 pGVM->gmm.s.Stats.Allocated.cBasePages -= cPages;
3175 pGVM->gmm.s.Stats.cPrivatePages -= cPages;
3176 pGMM->cAllocatedPages -= cPages;
3177 }
3178 else
3179 rc = VERR_GMM_PAGE_NOT_FOUND;
3180 }
3181 else
3182 rc = VERR_GMM_IS_NOT_SANE;
3183
3184 gmmR0MutexRelease(pGMM);
3185 LogFlow(("GMMR0FreeLargePage: returns %Rrc\n", rc));
3186 return rc;
3187}
3188
3189
3190/**
3191 * VMMR0 request wrapper for GMMR0FreeLargePage.
3192 *
3193 * @returns see GMMR0FreeLargePage.
3194 * @param pVM The cross context VM structure.
3195 * @param idCpu The VCPU id.
3196 * @param pReq Pointer to the request packet.
3197 */
3198GMMR0DECL(int) GMMR0FreeLargePageReq(PVM pVM, VMCPUID idCpu, PGMMFREELARGEPAGEREQ pReq)
3199{
3200 /*
3201 * Validate input and pass it on.
3202 */
3203 AssertPtrReturn(pVM, VERR_INVALID_POINTER);
3204 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
3205 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(GMMFREEPAGESREQ),
3206 ("%#x != %#x\n", pReq->Hdr.cbReq, sizeof(GMMFREEPAGESREQ)),
3207 VERR_INVALID_PARAMETER);
3208
3209 return GMMR0FreeLargePage(pVM, idCpu, pReq->idPage);
3210}
3211
3212
3213/**
3214 * Frees a chunk, giving it back to the host OS.
3215 *
3216 * @param pGMM Pointer to the GMM instance.
3217 * @param pGVM This is set when called from GMMR0CleanupVM so we can
3218 * unmap and free the chunk in one go.
3219 * @param pChunk The chunk to free.
3220 * @param fRelaxedSem Whether we can release the semaphore while doing the
3221 * freeing (@c true) or not.
3222 */
3223static bool gmmR0FreeChunk(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk, bool fRelaxedSem)
3224{
3225 Assert(pChunk->Core.Key != NIL_GMM_CHUNKID);
3226
3227 GMMR0CHUNKMTXSTATE MtxState;
3228 gmmR0ChunkMutexAcquire(&MtxState, pGMM, pChunk, GMMR0CHUNK_MTX_KEEP_GIANT);
3229
3230 /*
3231 * Cleanup hack! Unmap the chunk from the callers address space.
3232 * This shouldn't happen, so screw lock contention...
3233 */
3234 if ( pChunk->cMappingsX
3235 && !pGMM->fLegacyAllocationMode
3236 && pGVM)
3237 gmmR0UnmapChunkLocked(pGMM, pGVM, pChunk);
3238
3239 /*
3240 * If there are current mappings of the chunk, then request the
3241 * VMs to unmap them. Reposition the chunk in the free list so
3242 * it won't be a likely candidate for allocations.
3243 */
3244 if (pChunk->cMappingsX)
3245 {
3246 /** @todo R0 -> VM request */
3247 /* The chunk can be mapped by more than one VM if fBoundMemoryMode is false! */
3248 Log(("gmmR0FreeChunk: chunk still has %d mappings; don't free!\n", pChunk->cMappingsX));
3249 gmmR0ChunkMutexRelease(&MtxState, pChunk);
3250 return false;
3251 }
3252
3253
3254 /*
3255 * Save and trash the handle.
3256 */
3257 RTR0MEMOBJ const hMemObj = pChunk->hMemObj;
3258 pChunk->hMemObj = NIL_RTR0MEMOBJ;
3259
3260 /*
3261 * Unlink it from everywhere.
3262 */
3263 gmmR0UnlinkChunk(pChunk);
3264
3265 RTListNodeRemove(&pChunk->ListNode);
3266
3267 PAVLU32NODECORE pCore = RTAvlU32Remove(&pGMM->pChunks, pChunk->Core.Key);
3268 Assert(pCore == &pChunk->Core); NOREF(pCore);
3269
3270 PGMMCHUNKTLBE pTlbe = &pGMM->ChunkTLB.aEntries[GMM_CHUNKTLB_IDX(pChunk->Core.Key)];
3271 if (pTlbe->pChunk == pChunk)
3272 {
3273 pTlbe->idChunk = NIL_GMM_CHUNKID;
3274 pTlbe->pChunk = NULL;
3275 }
3276
3277 Assert(pGMM->cChunks > 0);
3278 pGMM->cChunks--;
3279
3280 /*
3281 * Free the Chunk ID before dropping the locks and freeing the rest.
3282 */
3283 gmmR0FreeChunkId(pGMM, pChunk->Core.Key);
3284 pChunk->Core.Key = NIL_GMM_CHUNKID;
3285
3286 pGMM->cFreedChunks++;
3287
3288 gmmR0ChunkMutexRelease(&MtxState, NULL);
3289 if (fRelaxedSem)
3290 gmmR0MutexRelease(pGMM);
3291
3292 RTMemFree(pChunk->paMappingsX);
3293 pChunk->paMappingsX = NULL;
3294
3295 RTMemFree(pChunk);
3296
3297 int rc = RTR0MemObjFree(hMemObj, false /* fFreeMappings */);
3298 AssertLogRelRC(rc);
3299
3300 if (fRelaxedSem)
3301 gmmR0MutexAcquire(pGMM);
3302 return fRelaxedSem;
3303}
3304
3305
3306/**
3307 * Free page worker.
3308 *
3309 * The caller does all the statistic decrementing, we do all the incrementing.
3310 *
3311 * @param pGMM Pointer to the GMM instance data.
3312 * @param pGVM Pointer to the GVM instance.
3313 * @param pChunk Pointer to the chunk this page belongs to.
3314 * @param idPage The Page ID.
3315 * @param pPage Pointer to the page.
3316 */
3317static void gmmR0FreePageWorker(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk, uint32_t idPage, PGMMPAGE pPage)
3318{
3319 Log3(("F pPage=%p iPage=%#x/%#x u2State=%d iFreeHead=%#x\n",
3320 pPage, pPage - &pChunk->aPages[0], idPage, pPage->Common.u2State, pChunk->iFreeHead)); NOREF(idPage);
3321
3322 /*
3323 * Put the page on the free list.
3324 */
3325 pPage->u = 0;
3326 pPage->Free.u2State = GMM_PAGE_STATE_FREE;
3327 Assert(pChunk->iFreeHead < RT_ELEMENTS(pChunk->aPages) || pChunk->iFreeHead == UINT16_MAX);
3328 pPage->Free.iNext = pChunk->iFreeHead;
3329 pChunk->iFreeHead = pPage - &pChunk->aPages[0];
3330
3331 /*
3332 * Update statistics (the cShared/cPrivate stats are up to date already),
3333 * and relink the chunk if necessary.
3334 */
3335 unsigned const cFree = pChunk->cFree;
3336 if ( !cFree
3337 || gmmR0SelectFreeSetList(cFree) != gmmR0SelectFreeSetList(cFree + 1))
3338 {
3339 gmmR0UnlinkChunk(pChunk);
3340 pChunk->cFree++;
3341 gmmR0SelectSetAndLinkChunk(pGMM, pGVM, pChunk);
3342 }
3343 else
3344 {
3345 pChunk->cFree = cFree + 1;
3346 pChunk->pSet->cFreePages++;
3347 }
3348
3349 /*
3350 * If the chunk becomes empty, consider giving memory back to the host OS.
3351 *
3352 * The current strategy is to try give it back if there are other chunks
3353 * in this free list, meaning if there are at least 240 free pages in this
3354 * category. Note that since there are probably mappings of the chunk,
3355 * it won't be freed up instantly, which probably screws up this logic
3356 * a bit...
3357 */
3358 /** @todo Do this on the way out. */
3359 if (RT_UNLIKELY( pChunk->cFree == GMM_CHUNK_NUM_PAGES
3360 && pChunk->pFreeNext
3361 && pChunk->pFreePrev /** @todo this is probably misfiring, see reset... */
3362 && !pGMM->fLegacyAllocationMode))
3363 gmmR0FreeChunk(pGMM, NULL, pChunk, false);
3364
3365}
3366
3367
3368/**
3369 * Frees a shared page, the page is known to exist and be valid and such.
3370 *
3371 * @param pGMM Pointer to the GMM instance.
3372 * @param pGVM Pointer to the GVM instance.
3373 * @param idPage The page id.
3374 * @param pPage The page structure.
3375 */
3376DECLINLINE(void) gmmR0FreeSharedPage(PGMM pGMM, PGVM pGVM, uint32_t idPage, PGMMPAGE pPage)
3377{
3378 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, idPage >> GMM_CHUNKID_SHIFT);
3379 Assert(pChunk);
3380 Assert(pChunk->cFree < GMM_CHUNK_NUM_PAGES);
3381 Assert(pChunk->cShared > 0);
3382 Assert(pGMM->cSharedPages > 0);
3383 Assert(pGMM->cAllocatedPages > 0);
3384 Assert(!pPage->Shared.cRefs);
3385
3386 pChunk->cShared--;
3387 pGMM->cAllocatedPages--;
3388 pGMM->cSharedPages--;
3389 gmmR0FreePageWorker(pGMM, pGVM, pChunk, idPage, pPage);
3390}
3391
3392
3393/**
3394 * Frees a private page, the page is known to exist and be valid and such.
3395 *
3396 * @param pGMM Pointer to the GMM instance.
3397 * @param pGVM Pointer to the GVM instance.
3398 * @param idPage The page id.
3399 * @param pPage The page structure.
3400 */
3401DECLINLINE(void) gmmR0FreePrivatePage(PGMM pGMM, PGVM pGVM, uint32_t idPage, PGMMPAGE pPage)
3402{
3403 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, idPage >> GMM_CHUNKID_SHIFT);
3404 Assert(pChunk);
3405 Assert(pChunk->cFree < GMM_CHUNK_NUM_PAGES);
3406 Assert(pChunk->cPrivate > 0);
3407 Assert(pGMM->cAllocatedPages > 0);
3408
3409 pChunk->cPrivate--;
3410 pGMM->cAllocatedPages--;
3411 gmmR0FreePageWorker(pGMM, pGVM, pChunk, idPage, pPage);
3412}
3413
3414
3415/**
3416 * Common worker for GMMR0FreePages and GMMR0BalloonedPages.
3417 *
3418 * @returns VBox status code:
3419 * @retval xxx
3420 *
3421 * @param pGMM Pointer to the GMM instance data.
3422 * @param pGVM Pointer to the VM.
3423 * @param cPages The number of pages to free.
3424 * @param paPages Pointer to the page descriptors.
3425 * @param enmAccount The account this relates to.
3426 */
3427static int gmmR0FreePages(PGMM pGMM, PGVM pGVM, uint32_t cPages, PGMMFREEPAGEDESC paPages, GMMACCOUNT enmAccount)
3428{
3429 /*
3430 * Check that the request isn't impossible wrt to the account status.
3431 */
3432 switch (enmAccount)
3433 {
3434 case GMMACCOUNT_BASE:
3435 if (RT_UNLIKELY(pGVM->gmm.s.Stats.Allocated.cBasePages < cPages))
3436 {
3437 Log(("gmmR0FreePages: allocated=%#llx cPages=%#x!\n", pGVM->gmm.s.Stats.Allocated.cBasePages, cPages));
3438 return VERR_GMM_ATTEMPT_TO_FREE_TOO_MUCH;
3439 }
3440 break;
3441 case GMMACCOUNT_SHADOW:
3442 if (RT_UNLIKELY(pGVM->gmm.s.Stats.Allocated.cShadowPages < cPages))
3443 {
3444 Log(("gmmR0FreePages: allocated=%#llx cPages=%#x!\n", pGVM->gmm.s.Stats.Allocated.cShadowPages, cPages));
3445 return VERR_GMM_ATTEMPT_TO_FREE_TOO_MUCH;
3446 }
3447 break;
3448 case GMMACCOUNT_FIXED:
3449 if (RT_UNLIKELY(pGVM->gmm.s.Stats.Allocated.cFixedPages < cPages))
3450 {
3451 Log(("gmmR0FreePages: allocated=%#llx cPages=%#x!\n", pGVM->gmm.s.Stats.Allocated.cFixedPages, cPages));
3452 return VERR_GMM_ATTEMPT_TO_FREE_TOO_MUCH;
3453 }
3454 break;
3455 default:
3456 AssertMsgFailedReturn(("enmAccount=%d\n", enmAccount), VERR_IPE_NOT_REACHED_DEFAULT_CASE);
3457 }
3458
3459 /*
3460 * Walk the descriptors and free the pages.
3461 *
3462 * Statistics (except the account) are being updated as we go along,
3463 * unlike the alloc code. Also, stop on the first error.
3464 */
3465 int rc = VINF_SUCCESS;
3466 uint32_t iPage;
3467 for (iPage = 0; iPage < cPages; iPage++)
3468 {
3469 uint32_t idPage = paPages[iPage].idPage;
3470 PGMMPAGE pPage = gmmR0GetPage(pGMM, idPage);
3471 if (RT_LIKELY(pPage))
3472 {
3473 if (RT_LIKELY(GMM_PAGE_IS_PRIVATE(pPage)))
3474 {
3475 if (RT_LIKELY(pPage->Private.hGVM == pGVM->hSelf))
3476 {
3477 Assert(pGVM->gmm.s.Stats.cPrivatePages);
3478 pGVM->gmm.s.Stats.cPrivatePages--;
3479 gmmR0FreePrivatePage(pGMM, pGVM, idPage, pPage);
3480 }
3481 else
3482 {
3483 Log(("gmmR0AllocatePages: #%#x/%#x: not owner! hGVM=%#x hSelf=%#x\n", iPage, idPage,
3484 pPage->Private.hGVM, pGVM->hSelf));
3485 rc = VERR_GMM_NOT_PAGE_OWNER;
3486 break;
3487 }
3488 }
3489 else if (RT_LIKELY(GMM_PAGE_IS_SHARED(pPage)))
3490 {
3491 Assert(pGVM->gmm.s.Stats.cSharedPages);
3492 Assert(pPage->Shared.cRefs);
3493#if defined(VBOX_WITH_PAGE_SHARING) && defined(VBOX_STRICT) && HC_ARCH_BITS == 64
3494 if (pPage->Shared.u14Checksum)
3495 {
3496 uint32_t uChecksum = gmmR0StrictPageChecksum(pGMM, pGVM, idPage);
3497 uChecksum &= UINT32_C(0x00003fff);
3498 AssertMsg(!uChecksum || uChecksum == pPage->Shared.u14Checksum,
3499 ("%#x vs %#x - idPage=%#x\n", uChecksum, pPage->Shared.u14Checksum, idPage));
3500 }
3501#endif
3502 pGVM->gmm.s.Stats.cSharedPages--;
3503 if (!--pPage->Shared.cRefs)
3504 gmmR0FreeSharedPage(pGMM, pGVM, idPage, pPage);
3505 else
3506 {
3507 Assert(pGMM->cDuplicatePages);
3508 pGMM->cDuplicatePages--;
3509 }
3510 }
3511 else
3512 {
3513 Log(("gmmR0AllocatePages: #%#x/%#x: already free!\n", iPage, idPage));
3514 rc = VERR_GMM_PAGE_ALREADY_FREE;
3515 break;
3516 }
3517 }
3518 else
3519 {
3520 Log(("gmmR0AllocatePages: #%#x/%#x: not found!\n", iPage, idPage));
3521 rc = VERR_GMM_PAGE_NOT_FOUND;
3522 break;
3523 }
3524 paPages[iPage].idPage = NIL_GMM_PAGEID;
3525 }
3526
3527 /*
3528 * Update the account.
3529 */
3530 switch (enmAccount)
3531 {
3532 case GMMACCOUNT_BASE: pGVM->gmm.s.Stats.Allocated.cBasePages -= iPage; break;
3533 case GMMACCOUNT_SHADOW: pGVM->gmm.s.Stats.Allocated.cShadowPages -= iPage; break;
3534 case GMMACCOUNT_FIXED: pGVM->gmm.s.Stats.Allocated.cFixedPages -= iPage; break;
3535 default:
3536 AssertMsgFailedReturn(("enmAccount=%d\n", enmAccount), VERR_IPE_NOT_REACHED_DEFAULT_CASE);
3537 }
3538
3539 /*
3540 * Any threshold stuff to be done here?
3541 */
3542
3543 return rc;
3544}
3545
3546
3547/**
3548 * Free one or more pages.
3549 *
3550 * This is typically used at reset time or power off.
3551 *
3552 * @returns VBox status code:
3553 * @retval xxx
3554 *
3555 * @param pVM The cross context VM structure.
3556 * @param idCpu The VCPU id.
3557 * @param cPages The number of pages to allocate.
3558 * @param paPages Pointer to the page descriptors containing the page IDs
3559 * for each page.
3560 * @param enmAccount The account this relates to.
3561 * @thread EMT.
3562 */
3563GMMR0DECL(int) GMMR0FreePages(PVM pVM, VMCPUID idCpu, uint32_t cPages, PGMMFREEPAGEDESC paPages, GMMACCOUNT enmAccount)
3564{
3565 LogFlow(("GMMR0FreePages: pVM=%p cPages=%#x paPages=%p enmAccount=%d\n", pVM, cPages, paPages, enmAccount));
3566
3567 /*
3568 * Validate input and get the basics.
3569 */
3570 PGMM pGMM;
3571 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
3572 PGVM pGVM;
3573 int rc = GVMMR0ByVMAndEMT(pVM, idCpu, &pGVM);
3574 if (RT_FAILURE(rc))
3575 return rc;
3576
3577 AssertPtrReturn(paPages, VERR_INVALID_PARAMETER);
3578 AssertMsgReturn(enmAccount > GMMACCOUNT_INVALID && enmAccount < GMMACCOUNT_END, ("%d\n", enmAccount), VERR_INVALID_PARAMETER);
3579 AssertMsgReturn(cPages > 0 && cPages < RT_BIT(32 - PAGE_SHIFT), ("%#x\n", cPages), VERR_INVALID_PARAMETER);
3580
3581 for (unsigned iPage = 0; iPage < cPages; iPage++)
3582 AssertMsgReturn( paPages[iPage].idPage <= GMM_PAGEID_LAST
3583 /*|| paPages[iPage].idPage == NIL_GMM_PAGEID*/,
3584 ("#%#x: %#x\n", iPage, paPages[iPage].idPage), VERR_INVALID_PARAMETER);
3585
3586 /*
3587 * Take the semaphore and call the worker function.
3588 */
3589 gmmR0MutexAcquire(pGMM);
3590 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
3591 {
3592 rc = gmmR0FreePages(pGMM, pGVM, cPages, paPages, enmAccount);
3593 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
3594 }
3595 else
3596 rc = VERR_GMM_IS_NOT_SANE;
3597 gmmR0MutexRelease(pGMM);
3598 LogFlow(("GMMR0FreePages: returns %Rrc\n", rc));
3599 return rc;
3600}
3601
3602
3603/**
3604 * VMMR0 request wrapper for GMMR0FreePages.
3605 *
3606 * @returns see GMMR0FreePages.
3607 * @param pVM The cross context VM structure.
3608 * @param idCpu The VCPU id.
3609 * @param pReq Pointer to the request packet.
3610 */
3611GMMR0DECL(int) GMMR0FreePagesReq(PVM pVM, VMCPUID idCpu, PGMMFREEPAGESREQ pReq)
3612{
3613 /*
3614 * Validate input and pass it on.
3615 */
3616 AssertPtrReturn(pVM, VERR_INVALID_POINTER);
3617 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
3618 AssertMsgReturn(pReq->Hdr.cbReq >= RT_UOFFSETOF(GMMFREEPAGESREQ, aPages[0]),
3619 ("%#x < %#x\n", pReq->Hdr.cbReq, RT_UOFFSETOF(GMMFREEPAGESREQ, aPages[0])),
3620 VERR_INVALID_PARAMETER);
3621 AssertMsgReturn(pReq->Hdr.cbReq == RT_UOFFSETOF(GMMFREEPAGESREQ, aPages[pReq->cPages]),
3622 ("%#x != %#x\n", pReq->Hdr.cbReq, RT_UOFFSETOF(GMMFREEPAGESREQ, aPages[pReq->cPages])),
3623 VERR_INVALID_PARAMETER);
3624
3625 return GMMR0FreePages(pVM, idCpu, pReq->cPages, &pReq->aPages[0], pReq->enmAccount);
3626}
3627
3628
3629/**
3630 * Report back on a memory ballooning request.
3631 *
3632 * The request may or may not have been initiated by the GMM. If it was initiated
3633 * by the GMM it is important that this function is called even if no pages were
3634 * ballooned.
3635 *
3636 * @returns VBox status code:
3637 * @retval VERR_GMM_ATTEMPT_TO_FREE_TOO_MUCH
3638 * @retval VERR_GMM_ATTEMPT_TO_DEFLATE_TOO_MUCH
3639 * @retval VERR_GMM_OVERCOMMITTED_TRY_AGAIN_IN_A_BIT - reset condition
3640 * indicating that we won't necessarily have sufficient RAM to boot
3641 * the VM again and that it should pause until this changes (we'll try
3642 * balloon some other VM). (For standard deflate we have little choice
3643 * but to hope the VM won't use the memory that was returned to it.)
3644 *
3645 * @param pVM The cross context VM structure.
3646 * @param idCpu The VCPU id.
3647 * @param enmAction Inflate/deflate/reset.
3648 * @param cBalloonedPages The number of pages that was ballooned.
3649 *
3650 * @thread EMT.
3651 */
3652GMMR0DECL(int) GMMR0BalloonedPages(PVM pVM, VMCPUID idCpu, GMMBALLOONACTION enmAction, uint32_t cBalloonedPages)
3653{
3654 LogFlow(("GMMR0BalloonedPages: pVM=%p enmAction=%d cBalloonedPages=%#x\n",
3655 pVM, enmAction, cBalloonedPages));
3656
3657 AssertMsgReturn(cBalloonedPages < RT_BIT(32 - PAGE_SHIFT), ("%#x\n", cBalloonedPages), VERR_INVALID_PARAMETER);
3658
3659 /*
3660 * Validate input and get the basics.
3661 */
3662 PGMM pGMM;
3663 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
3664 PGVM pGVM;
3665 int rc = GVMMR0ByVMAndEMT(pVM, idCpu, &pGVM);
3666 if (RT_FAILURE(rc))
3667 return rc;
3668
3669 /*
3670 * Take the semaphore and do some more validations.
3671 */
3672 gmmR0MutexAcquire(pGMM);
3673 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
3674 {
3675 switch (enmAction)
3676 {
3677 case GMMBALLOONACTION_INFLATE:
3678 {
3679 if (RT_LIKELY(pGVM->gmm.s.Stats.Allocated.cBasePages + pGVM->gmm.s.Stats.cBalloonedPages + cBalloonedPages
3680 <= pGVM->gmm.s.Stats.Reserved.cBasePages))
3681 {
3682 /*
3683 * Record the ballooned memory.
3684 */
3685 pGMM->cBalloonedPages += cBalloonedPages;
3686 if (pGVM->gmm.s.Stats.cReqBalloonedPages)
3687 {
3688 /* Codepath never taken. Might be interesting in the future to request ballooned memory from guests in low memory conditions.. */
3689 AssertFailed();
3690
3691 pGVM->gmm.s.Stats.cBalloonedPages += cBalloonedPages;
3692 pGVM->gmm.s.Stats.cReqActuallyBalloonedPages += cBalloonedPages;
3693 Log(("GMMR0BalloonedPages: +%#x - Global=%#llx / VM: Total=%#llx Req=%#llx Actual=%#llx (pending)\n",
3694 cBalloonedPages, pGMM->cBalloonedPages, pGVM->gmm.s.Stats.cBalloonedPages,
3695 pGVM->gmm.s.Stats.cReqBalloonedPages, pGVM->gmm.s.Stats.cReqActuallyBalloonedPages));
3696 }
3697 else
3698 {
3699 pGVM->gmm.s.Stats.cBalloonedPages += cBalloonedPages;
3700 Log(("GMMR0BalloonedPages: +%#x - Global=%#llx / VM: Total=%#llx (user)\n",
3701 cBalloonedPages, pGMM->cBalloonedPages, pGVM->gmm.s.Stats.cBalloonedPages));
3702 }
3703 }
3704 else
3705 {
3706 Log(("GMMR0BalloonedPages: cBasePages=%#llx Total=%#llx cBalloonedPages=%#llx Reserved=%#llx\n",
3707 pGVM->gmm.s.Stats.Allocated.cBasePages, pGVM->gmm.s.Stats.cBalloonedPages, cBalloonedPages,
3708 pGVM->gmm.s.Stats.Reserved.cBasePages));
3709 rc = VERR_GMM_ATTEMPT_TO_FREE_TOO_MUCH;
3710 }
3711 break;
3712 }
3713
3714 case GMMBALLOONACTION_DEFLATE:
3715 {
3716 /* Deflate. */
3717 if (pGVM->gmm.s.Stats.cBalloonedPages >= cBalloonedPages)
3718 {
3719 /*
3720 * Record the ballooned memory.
3721 */
3722 Assert(pGMM->cBalloonedPages >= cBalloonedPages);
3723 pGMM->cBalloonedPages -= cBalloonedPages;
3724 pGVM->gmm.s.Stats.cBalloonedPages -= cBalloonedPages;
3725 if (pGVM->gmm.s.Stats.cReqDeflatePages)
3726 {
3727 AssertFailed(); /* This is path is for later. */
3728 Log(("GMMR0BalloonedPages: -%#x - Global=%#llx / VM: Total=%#llx Req=%#llx\n",
3729 cBalloonedPages, pGMM->cBalloonedPages, pGVM->gmm.s.Stats.cBalloonedPages, pGVM->gmm.s.Stats.cReqDeflatePages));
3730
3731 /*
3732 * Anything we need to do here now when the request has been completed?
3733 */
3734 pGVM->gmm.s.Stats.cReqDeflatePages = 0;
3735 }
3736 else
3737 Log(("GMMR0BalloonedPages: -%#x - Global=%#llx / VM: Total=%#llx (user)\n",
3738 cBalloonedPages, pGMM->cBalloonedPages, pGVM->gmm.s.Stats.cBalloonedPages));
3739 }
3740 else
3741 {
3742 Log(("GMMR0BalloonedPages: Total=%#llx cBalloonedPages=%#llx\n", pGVM->gmm.s.Stats.cBalloonedPages, cBalloonedPages));
3743 rc = VERR_GMM_ATTEMPT_TO_DEFLATE_TOO_MUCH;
3744 }
3745 break;
3746 }
3747
3748 case GMMBALLOONACTION_RESET:
3749 {
3750 /* Reset to an empty balloon. */
3751 Assert(pGMM->cBalloonedPages >= pGVM->gmm.s.Stats.cBalloonedPages);
3752
3753 pGMM->cBalloonedPages -= pGVM->gmm.s.Stats.cBalloonedPages;
3754 pGVM->gmm.s.Stats.cBalloonedPages = 0;
3755 break;
3756 }
3757
3758 default:
3759 rc = VERR_INVALID_PARAMETER;
3760 break;
3761 }
3762 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
3763 }
3764 else
3765 rc = VERR_GMM_IS_NOT_SANE;
3766
3767 gmmR0MutexRelease(pGMM);
3768 LogFlow(("GMMR0BalloonedPages: returns %Rrc\n", rc));
3769 return rc;
3770}
3771
3772
3773/**
3774 * VMMR0 request wrapper for GMMR0BalloonedPages.
3775 *
3776 * @returns see GMMR0BalloonedPages.
3777 * @param pVM The cross context VM structure.
3778 * @param idCpu The VCPU id.
3779 * @param pReq Pointer to the request packet.
3780 */
3781GMMR0DECL(int) GMMR0BalloonedPagesReq(PVM pVM, VMCPUID idCpu, PGMMBALLOONEDPAGESREQ pReq)
3782{
3783 /*
3784 * Validate input and pass it on.
3785 */
3786 AssertPtrReturn(pVM, VERR_INVALID_POINTER);
3787 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
3788 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(GMMBALLOONEDPAGESREQ),
3789 ("%#x < %#x\n", pReq->Hdr.cbReq, sizeof(GMMBALLOONEDPAGESREQ)),
3790 VERR_INVALID_PARAMETER);
3791
3792 return GMMR0BalloonedPages(pVM, idCpu, pReq->enmAction, pReq->cBalloonedPages);
3793}
3794
3795/**
3796 * Return memory statistics for the hypervisor
3797 *
3798 * @returns VBox status code:
3799 * @param pVM The cross context VM structure.
3800 * @param pReq Pointer to the request packet.
3801 */
3802GMMR0DECL(int) GMMR0QueryHypervisorMemoryStatsReq(PVM pVM, PGMMMEMSTATSREQ pReq)
3803{
3804 /*
3805 * Validate input and pass it on.
3806 */
3807 AssertPtrReturn(pVM, VERR_INVALID_POINTER);
3808 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
3809 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(GMMMEMSTATSREQ),
3810 ("%#x < %#x\n", pReq->Hdr.cbReq, sizeof(GMMMEMSTATSREQ)),
3811 VERR_INVALID_PARAMETER);
3812
3813 /*
3814 * Validate input and get the basics.
3815 */
3816 PGMM pGMM;
3817 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
3818 pReq->cAllocPages = pGMM->cAllocatedPages;
3819 pReq->cFreePages = (pGMM->cChunks << (GMM_CHUNK_SHIFT- PAGE_SHIFT)) - pGMM->cAllocatedPages;
3820 pReq->cBalloonedPages = pGMM->cBalloonedPages;
3821 pReq->cMaxPages = pGMM->cMaxPages;
3822 pReq->cSharedPages = pGMM->cDuplicatePages;
3823 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
3824
3825 return VINF_SUCCESS;
3826}
3827
3828/**
3829 * Return memory statistics for the VM
3830 *
3831 * @returns VBox status code:
3832 * @param pVM The cross context VM structure.
3833 * @param idCpu Cpu id.
3834 * @param pReq Pointer to the request packet.
3835 */
3836GMMR0DECL(int) GMMR0QueryMemoryStatsReq(PVM pVM, VMCPUID idCpu, PGMMMEMSTATSREQ pReq)
3837{
3838 /*
3839 * Validate input and pass it on.
3840 */
3841 AssertPtrReturn(pVM, VERR_INVALID_POINTER);
3842 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
3843 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(GMMMEMSTATSREQ),
3844 ("%#x < %#x\n", pReq->Hdr.cbReq, sizeof(GMMMEMSTATSREQ)),
3845 VERR_INVALID_PARAMETER);
3846
3847 /*
3848 * Validate input and get the basics.
3849 */
3850 PGMM pGMM;
3851 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
3852 PGVM pGVM;
3853 int rc = GVMMR0ByVMAndEMT(pVM, idCpu, &pGVM);
3854 if (RT_FAILURE(rc))
3855 return rc;
3856
3857 /*
3858 * Take the semaphore and do some more validations.
3859 */
3860 gmmR0MutexAcquire(pGMM);
3861 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
3862 {
3863 pReq->cAllocPages = pGVM->gmm.s.Stats.Allocated.cBasePages;
3864 pReq->cBalloonedPages = pGVM->gmm.s.Stats.cBalloonedPages;
3865 pReq->cMaxPages = pGVM->gmm.s.Stats.Reserved.cBasePages;
3866 pReq->cFreePages = pReq->cMaxPages - pReq->cAllocPages;
3867 }
3868 else
3869 rc = VERR_GMM_IS_NOT_SANE;
3870
3871 gmmR0MutexRelease(pGMM);
3872 LogFlow(("GMMR3QueryVMMemoryStats: returns %Rrc\n", rc));
3873 return rc;
3874}
3875
3876
3877/**
3878 * Worker for gmmR0UnmapChunk and gmmr0FreeChunk.
3879 *
3880 * Don't call this in legacy allocation mode!
3881 *
3882 * @returns VBox status code.
3883 * @param pGMM Pointer to the GMM instance data.
3884 * @param pGVM Pointer to the Global VM structure.
3885 * @param pChunk Pointer to the chunk to be unmapped.
3886 */
3887static int gmmR0UnmapChunkLocked(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk)
3888{
3889 Assert(!pGMM->fLegacyAllocationMode); NOREF(pGMM);
3890
3891 /*
3892 * Find the mapping and try unmapping it.
3893 */
3894 uint32_t cMappings = pChunk->cMappingsX;
3895 for (uint32_t i = 0; i < cMappings; i++)
3896 {
3897 Assert(pChunk->paMappingsX[i].pGVM && pChunk->paMappingsX[i].hMapObj != NIL_RTR0MEMOBJ);
3898 if (pChunk->paMappingsX[i].pGVM == pGVM)
3899 {
3900 /* unmap */
3901 int rc = RTR0MemObjFree(pChunk->paMappingsX[i].hMapObj, false /* fFreeMappings (NA) */);
3902 if (RT_SUCCESS(rc))
3903 {
3904 /* update the record. */
3905 cMappings--;
3906 if (i < cMappings)
3907 pChunk->paMappingsX[i] = pChunk->paMappingsX[cMappings];
3908 pChunk->paMappingsX[cMappings].hMapObj = NIL_RTR0MEMOBJ;
3909 pChunk->paMappingsX[cMappings].pGVM = NULL;
3910 Assert(pChunk->cMappingsX - 1U == cMappings);
3911 pChunk->cMappingsX = cMappings;
3912 }
3913
3914 return rc;
3915 }
3916 }
3917
3918 Log(("gmmR0UnmapChunk: Chunk %#x is not mapped into pGVM=%p/%#x\n", pChunk->Core.Key, pGVM, pGVM->hSelf));
3919 return VERR_GMM_CHUNK_NOT_MAPPED;
3920}
3921
3922
3923/**
3924 * Unmaps a chunk previously mapped into the address space of the current process.
3925 *
3926 * @returns VBox status code.
3927 * @param pGMM Pointer to the GMM instance data.
3928 * @param pGVM Pointer to the Global VM structure.
3929 * @param pChunk Pointer to the chunk to be unmapped.
3930 * @param fRelaxedSem Whether we can release the semaphore while doing the
3931 * mapping (@c true) or not.
3932 */
3933static int gmmR0UnmapChunk(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk, bool fRelaxedSem)
3934{
3935 if (!pGMM->fLegacyAllocationMode)
3936 {
3937 /*
3938 * Lock the chunk and if possible leave the giant GMM lock.
3939 */
3940 GMMR0CHUNKMTXSTATE MtxState;
3941 int rc = gmmR0ChunkMutexAcquire(&MtxState, pGMM, pChunk,
3942 fRelaxedSem ? GMMR0CHUNK_MTX_RETAKE_GIANT : GMMR0CHUNK_MTX_KEEP_GIANT);
3943 if (RT_SUCCESS(rc))
3944 {
3945 rc = gmmR0UnmapChunkLocked(pGMM, pGVM, pChunk);
3946 gmmR0ChunkMutexRelease(&MtxState, pChunk);
3947 }
3948 return rc;
3949 }
3950
3951 if (pChunk->hGVM == pGVM->hSelf)
3952 return VINF_SUCCESS;
3953
3954 Log(("gmmR0UnmapChunk: Chunk %#x is not mapped into pGVM=%p/%#x (legacy)\n", pChunk->Core.Key, pGVM, pGVM->hSelf));
3955 return VERR_GMM_CHUNK_NOT_MAPPED;
3956}
3957
3958
3959/**
3960 * Worker for gmmR0MapChunk.
3961 *
3962 * @returns VBox status code.
3963 * @param pGMM Pointer to the GMM instance data.
3964 * @param pGVM Pointer to the Global VM structure.
3965 * @param pChunk Pointer to the chunk to be mapped.
3966 * @param ppvR3 Where to store the ring-3 address of the mapping.
3967 * In the VERR_GMM_CHUNK_ALREADY_MAPPED case, this will be
3968 * contain the address of the existing mapping.
3969 */
3970static int gmmR0MapChunkLocked(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk, PRTR3PTR ppvR3)
3971{
3972 /*
3973 * If we're in legacy mode this is simple.
3974 */
3975 if (pGMM->fLegacyAllocationMode)
3976 {
3977 if (pChunk->hGVM != pGVM->hSelf)
3978 {
3979 Log(("gmmR0MapChunk: chunk %#x is already mapped at %p!\n", pChunk->Core.Key, *ppvR3));
3980 return VERR_GMM_CHUNK_NOT_FOUND;
3981 }
3982
3983 *ppvR3 = RTR0MemObjAddressR3(pChunk->hMemObj);
3984 return VINF_SUCCESS;
3985 }
3986
3987 /*
3988 * Check to see if the chunk is already mapped.
3989 */
3990 for (uint32_t i = 0; i < pChunk->cMappingsX; i++)
3991 {
3992 Assert(pChunk->paMappingsX[i].pGVM && pChunk->paMappingsX[i].hMapObj != NIL_RTR0MEMOBJ);
3993 if (pChunk->paMappingsX[i].pGVM == pGVM)
3994 {
3995 *ppvR3 = RTR0MemObjAddressR3(pChunk->paMappingsX[i].hMapObj);
3996 Log(("gmmR0MapChunk: chunk %#x is already mapped at %p!\n", pChunk->Core.Key, *ppvR3));
3997#ifdef VBOX_WITH_PAGE_SHARING
3998 /* The ring-3 chunk cache can be out of sync; don't fail. */
3999 return VINF_SUCCESS;
4000#else
4001 return VERR_GMM_CHUNK_ALREADY_MAPPED;
4002#endif
4003 }
4004 }
4005
4006 /*
4007 * Do the mapping.
4008 */
4009 RTR0MEMOBJ hMapObj;
4010 int rc = RTR0MemObjMapUser(&hMapObj, pChunk->hMemObj, (RTR3PTR)-1, 0, RTMEM_PROT_READ | RTMEM_PROT_WRITE, NIL_RTR0PROCESS);
4011 if (RT_SUCCESS(rc))
4012 {
4013 /* reallocate the array? assumes few users per chunk (usually one). */
4014 unsigned iMapping = pChunk->cMappingsX;
4015 if ( iMapping <= 3
4016 || (iMapping & 3) == 0)
4017 {
4018 unsigned cNewSize = iMapping <= 3
4019 ? iMapping + 1
4020 : iMapping + 4;
4021 Assert(cNewSize < 4 || RT_ALIGN_32(cNewSize, 4) == cNewSize);
4022 if (RT_UNLIKELY(cNewSize > UINT16_MAX))
4023 {
4024 rc = RTR0MemObjFree(hMapObj, false /* fFreeMappings (NA) */); AssertRC(rc);
4025 return VERR_GMM_TOO_MANY_CHUNK_MAPPINGS;
4026 }
4027
4028 void *pvMappings = RTMemRealloc(pChunk->paMappingsX, cNewSize * sizeof(pChunk->paMappingsX[0]));
4029 if (RT_UNLIKELY(!pvMappings))
4030 {
4031 rc = RTR0MemObjFree(hMapObj, false /* fFreeMappings (NA) */); AssertRC(rc);
4032 return VERR_NO_MEMORY;
4033 }
4034 pChunk->paMappingsX = (PGMMCHUNKMAP)pvMappings;
4035 }
4036
4037 /* insert new entry */
4038 pChunk->paMappingsX[iMapping].hMapObj = hMapObj;
4039 pChunk->paMappingsX[iMapping].pGVM = pGVM;
4040 Assert(pChunk->cMappingsX == iMapping);
4041 pChunk->cMappingsX = iMapping + 1;
4042
4043 *ppvR3 = RTR0MemObjAddressR3(hMapObj);
4044 }
4045
4046 return rc;
4047}
4048
4049
4050/**
4051 * Maps a chunk into the user address space of the current process.
4052 *
4053 * @returns VBox status code.
4054 * @param pGMM Pointer to the GMM instance data.
4055 * @param pGVM Pointer to the Global VM structure.
4056 * @param pChunk Pointer to the chunk to be mapped.
4057 * @param fRelaxedSem Whether we can release the semaphore while doing the
4058 * mapping (@c true) or not.
4059 * @param ppvR3 Where to store the ring-3 address of the mapping.
4060 * In the VERR_GMM_CHUNK_ALREADY_MAPPED case, this will be
4061 * contain the address of the existing mapping.
4062 */
4063static int gmmR0MapChunk(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk, bool fRelaxedSem, PRTR3PTR ppvR3)
4064{
4065 /*
4066 * Take the chunk lock and leave the giant GMM lock when possible, then
4067 * call the worker function.
4068 */
4069 GMMR0CHUNKMTXSTATE MtxState;
4070 int rc = gmmR0ChunkMutexAcquire(&MtxState, pGMM, pChunk,
4071 fRelaxedSem ? GMMR0CHUNK_MTX_RETAKE_GIANT : GMMR0CHUNK_MTX_KEEP_GIANT);
4072 if (RT_SUCCESS(rc))
4073 {
4074 rc = gmmR0MapChunkLocked(pGMM, pGVM, pChunk, ppvR3);
4075 gmmR0ChunkMutexRelease(&MtxState, pChunk);
4076 }
4077
4078 return rc;
4079}
4080
4081
4082
4083#if defined(VBOX_WITH_PAGE_SHARING) || (defined(VBOX_STRICT) && HC_ARCH_BITS == 64)
4084/**
4085 * Check if a chunk is mapped into the specified VM
4086 *
4087 * @returns mapped yes/no
4088 * @param pGMM Pointer to the GMM instance.
4089 * @param pGVM Pointer to the Global VM structure.
4090 * @param pChunk Pointer to the chunk to be mapped.
4091 * @param ppvR3 Where to store the ring-3 address of the mapping.
4092 */
4093static bool gmmR0IsChunkMapped(PGMM pGMM, PGVM pGVM, PGMMCHUNK pChunk, PRTR3PTR ppvR3)
4094{
4095 GMMR0CHUNKMTXSTATE MtxState;
4096 gmmR0ChunkMutexAcquire(&MtxState, pGMM, pChunk, GMMR0CHUNK_MTX_KEEP_GIANT);
4097 for (uint32_t i = 0; i < pChunk->cMappingsX; i++)
4098 {
4099 Assert(pChunk->paMappingsX[i].pGVM && pChunk->paMappingsX[i].hMapObj != NIL_RTR0MEMOBJ);
4100 if (pChunk->paMappingsX[i].pGVM == pGVM)
4101 {
4102 *ppvR3 = RTR0MemObjAddressR3(pChunk->paMappingsX[i].hMapObj);
4103 gmmR0ChunkMutexRelease(&MtxState, pChunk);
4104 return true;
4105 }
4106 }
4107 *ppvR3 = NULL;
4108 gmmR0ChunkMutexRelease(&MtxState, pChunk);
4109 return false;
4110}
4111#endif /* VBOX_WITH_PAGE_SHARING || (VBOX_STRICT && 64-BIT) */
4112
4113
4114/**
4115 * Map a chunk and/or unmap another chunk.
4116 *
4117 * The mapping and unmapping applies to the current process.
4118 *
4119 * This API does two things because it saves a kernel call per mapping when
4120 * when the ring-3 mapping cache is full.
4121 *
4122 * @returns VBox status code.
4123 * @param pVM The cross context VM structure.
4124 * @param idChunkMap The chunk to map. NIL_GMM_CHUNKID if nothing to map.
4125 * @param idChunkUnmap The chunk to unmap. NIL_GMM_CHUNKID if nothing to unmap.
4126 * @param ppvR3 Where to store the address of the mapped chunk. NULL is ok if nothing to map.
4127 * @thread EMT
4128 */
4129GMMR0DECL(int) GMMR0MapUnmapChunk(PVM pVM, uint32_t idChunkMap, uint32_t idChunkUnmap, PRTR3PTR ppvR3)
4130{
4131 LogFlow(("GMMR0MapUnmapChunk: pVM=%p idChunkMap=%#x idChunkUnmap=%#x ppvR3=%p\n",
4132 pVM, idChunkMap, idChunkUnmap, ppvR3));
4133
4134 /*
4135 * Validate input and get the basics.
4136 */
4137 PGMM pGMM;
4138 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
4139 PGVM pGVM;
4140 int rc = GVMMR0ByVM(pVM, &pGVM);
4141 if (RT_FAILURE(rc))
4142 return rc;
4143
4144 AssertCompile(NIL_GMM_CHUNKID == 0);
4145 AssertMsgReturn(idChunkMap <= GMM_CHUNKID_LAST, ("%#x\n", idChunkMap), VERR_INVALID_PARAMETER);
4146 AssertMsgReturn(idChunkUnmap <= GMM_CHUNKID_LAST, ("%#x\n", idChunkUnmap), VERR_INVALID_PARAMETER);
4147
4148 if ( idChunkMap == NIL_GMM_CHUNKID
4149 && idChunkUnmap == NIL_GMM_CHUNKID)
4150 return VERR_INVALID_PARAMETER;
4151
4152 if (idChunkMap != NIL_GMM_CHUNKID)
4153 {
4154 AssertPtrReturn(ppvR3, VERR_INVALID_POINTER);
4155 *ppvR3 = NIL_RTR3PTR;
4156 }
4157
4158 /*
4159 * Take the semaphore and do the work.
4160 *
4161 * The unmapping is done last since it's easier to undo a mapping than
4162 * undoing an unmapping. The ring-3 mapping cache cannot not be so big
4163 * that it pushes the user virtual address space to within a chunk of
4164 * it it's limits, so, no problem here.
4165 */
4166 gmmR0MutexAcquire(pGMM);
4167 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
4168 {
4169 PGMMCHUNK pMap = NULL;
4170 if (idChunkMap != NIL_GVM_HANDLE)
4171 {
4172 pMap = gmmR0GetChunk(pGMM, idChunkMap);
4173 if (RT_LIKELY(pMap))
4174 rc = gmmR0MapChunk(pGMM, pGVM, pMap, true /*fRelaxedSem*/, ppvR3);
4175 else
4176 {
4177 Log(("GMMR0MapUnmapChunk: idChunkMap=%#x\n", idChunkMap));
4178 rc = VERR_GMM_CHUNK_NOT_FOUND;
4179 }
4180 }
4181/** @todo split this operation, the bail out might (theoretcially) not be
4182 * entirely safe. */
4183
4184 if ( idChunkUnmap != NIL_GMM_CHUNKID
4185 && RT_SUCCESS(rc))
4186 {
4187 PGMMCHUNK pUnmap = gmmR0GetChunk(pGMM, idChunkUnmap);
4188 if (RT_LIKELY(pUnmap))
4189 rc = gmmR0UnmapChunk(pGMM, pGVM, pUnmap, true /*fRelaxedSem*/);
4190 else
4191 {
4192 Log(("GMMR0MapUnmapChunk: idChunkUnmap=%#x\n", idChunkUnmap));
4193 rc = VERR_GMM_CHUNK_NOT_FOUND;
4194 }
4195
4196 if (RT_FAILURE(rc) && pMap)
4197 gmmR0UnmapChunk(pGMM, pGVM, pMap, false /*fRelaxedSem*/);
4198 }
4199
4200 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
4201 }
4202 else
4203 rc = VERR_GMM_IS_NOT_SANE;
4204 gmmR0MutexRelease(pGMM);
4205
4206 LogFlow(("GMMR0MapUnmapChunk: returns %Rrc\n", rc));
4207 return rc;
4208}
4209
4210
4211/**
4212 * VMMR0 request wrapper for GMMR0MapUnmapChunk.
4213 *
4214 * @returns see GMMR0MapUnmapChunk.
4215 * @param pVM The cross context VM structure.
4216 * @param pReq Pointer to the request packet.
4217 */
4218GMMR0DECL(int) GMMR0MapUnmapChunkReq(PVM pVM, PGMMMAPUNMAPCHUNKREQ pReq)
4219{
4220 /*
4221 * Validate input and pass it on.
4222 */
4223 AssertPtrReturn(pVM, VERR_INVALID_POINTER);
4224 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
4225 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(*pReq), ("%#x != %#x\n", pReq->Hdr.cbReq, sizeof(*pReq)), VERR_INVALID_PARAMETER);
4226
4227 return GMMR0MapUnmapChunk(pVM, pReq->idChunkMap, pReq->idChunkUnmap, &pReq->pvR3);
4228}
4229
4230
4231/**
4232 * Legacy mode API for supplying pages.
4233 *
4234 * The specified user address points to a allocation chunk sized block that
4235 * will be locked down and used by the GMM when the GM asks for pages.
4236 *
4237 * @returns VBox status code.
4238 * @param pVM The cross context VM structure.
4239 * @param idCpu The VCPU id.
4240 * @param pvR3 Pointer to the chunk size memory block to lock down.
4241 */
4242GMMR0DECL(int) GMMR0SeedChunk(PVM pVM, VMCPUID idCpu, RTR3PTR pvR3)
4243{
4244 /*
4245 * Validate input and get the basics.
4246 */
4247 PGMM pGMM;
4248 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
4249 PGVM pGVM;
4250 int rc = GVMMR0ByVMAndEMT(pVM, idCpu, &pGVM);
4251 if (RT_FAILURE(rc))
4252 return rc;
4253
4254 AssertPtrReturn(pvR3, VERR_INVALID_POINTER);
4255 AssertReturn(!(PAGE_OFFSET_MASK & pvR3), VERR_INVALID_POINTER);
4256
4257 if (!pGMM->fLegacyAllocationMode)
4258 {
4259 Log(("GMMR0SeedChunk: not in legacy allocation mode!\n"));
4260 return VERR_NOT_SUPPORTED;
4261 }
4262
4263 /*
4264 * Lock the memory and add it as new chunk with our hGVM.
4265 * (The GMM locking is done inside gmmR0RegisterChunk.)
4266 */
4267 RTR0MEMOBJ MemObj;
4268 rc = RTR0MemObjLockUser(&MemObj, pvR3, GMM_CHUNK_SIZE, RTMEM_PROT_READ | RTMEM_PROT_WRITE, NIL_RTR0PROCESS);
4269 if (RT_SUCCESS(rc))
4270 {
4271 rc = gmmR0RegisterChunk(pGMM, &pGVM->gmm.s.Private, MemObj, pGVM->hSelf, 0 /*fChunkFlags*/, NULL);
4272 if (RT_SUCCESS(rc))
4273 gmmR0MutexRelease(pGMM);
4274 else
4275 RTR0MemObjFree(MemObj, false /* fFreeMappings */);
4276 }
4277
4278 LogFlow(("GMMR0SeedChunk: rc=%d (pvR3=%p)\n", rc, pvR3));
4279 return rc;
4280}
4281
4282#ifdef VBOX_WITH_PAGE_SHARING
4283
4284# ifdef VBOX_STRICT
4285/**
4286 * For checksumming shared pages in strict builds.
4287 *
4288 * The purpose is making sure that a page doesn't change.
4289 *
4290 * @returns Checksum, 0 on failure.
4291 * @param pGMM The GMM instance data.
4292 * @param pGVM Pointer to the kernel-only VM instace data.
4293 * @param idPage The page ID.
4294 */
4295static uint32_t gmmR0StrictPageChecksum(PGMM pGMM, PGVM pGVM, uint32_t idPage)
4296{
4297 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, idPage >> GMM_CHUNKID_SHIFT);
4298 AssertMsgReturn(pChunk, ("idPage=%#x\n", idPage), 0);
4299
4300 uint8_t *pbChunk;
4301 if (!gmmR0IsChunkMapped(pGMM, pGVM, pChunk, (PRTR3PTR)&pbChunk))
4302 return 0;
4303 uint8_t const *pbPage = pbChunk + ((idPage & GMM_PAGEID_IDX_MASK) << PAGE_SHIFT);
4304
4305 return RTCrc32(pbPage, PAGE_SIZE);
4306}
4307# endif /* VBOX_STRICT */
4308
4309
4310/**
4311 * Calculates the module hash value.
4312 *
4313 * @returns Hash value.
4314 * @param pszModuleName The module name.
4315 * @param pszVersion The module version string.
4316 */
4317static uint32_t gmmR0ShModCalcHash(const char *pszModuleName, const char *pszVersion)
4318{
4319 return RTStrHash1ExN(3, pszModuleName, RTSTR_MAX, "::", (size_t)2, pszVersion, RTSTR_MAX);
4320}
4321
4322
4323/**
4324 * Finds a global module.
4325 *
4326 * @returns Pointer to the global module on success, NULL if not found.
4327 * @param pGMM The GMM instance data.
4328 * @param uHash The hash as calculated by gmmR0ShModCalcHash.
4329 * @param cbModule The module size.
4330 * @param enmGuestOS The guest OS type.
4331 * @param cRegions The number of regions.
4332 * @param pszModuleName The module name.
4333 * @param pszVersion The module version.
4334 * @param paRegions The region descriptions.
4335 */
4336static PGMMSHAREDMODULE gmmR0ShModFindGlobal(PGMM pGMM, uint32_t uHash, uint32_t cbModule, VBOXOSFAMILY enmGuestOS,
4337 uint32_t cRegions, const char *pszModuleName, const char *pszVersion,
4338 struct VMMDEVSHAREDREGIONDESC const *paRegions)
4339{
4340 for (PGMMSHAREDMODULE pGblMod = (PGMMSHAREDMODULE)RTAvllU32Get(&pGMM->pGlobalSharedModuleTree, uHash);
4341 pGblMod;
4342 pGblMod = (PGMMSHAREDMODULE)pGblMod->Core.pList)
4343 {
4344 if (pGblMod->cbModule != cbModule)
4345 continue;
4346 if (pGblMod->enmGuestOS != enmGuestOS)
4347 continue;
4348 if (pGblMod->cRegions != cRegions)
4349 continue;
4350 if (strcmp(pGblMod->szName, pszModuleName))
4351 continue;
4352 if (strcmp(pGblMod->szVersion, pszVersion))
4353 continue;
4354
4355 uint32_t i;
4356 for (i = 0; i < cRegions; i++)
4357 {
4358 uint32_t off = paRegions[i].GCRegionAddr & PAGE_OFFSET_MASK;
4359 if (pGblMod->aRegions[i].off != off)
4360 break;
4361
4362 uint32_t cb = RT_ALIGN_32(paRegions[i].cbRegion + off, PAGE_SIZE);
4363 if (pGblMod->aRegions[i].cb != cb)
4364 break;
4365 }
4366
4367 if (i == cRegions)
4368 return pGblMod;
4369 }
4370
4371 return NULL;
4372}
4373
4374
4375/**
4376 * Creates a new global module.
4377 *
4378 * @returns VBox status code.
4379 * @param pGMM The GMM instance data.
4380 * @param uHash The hash as calculated by gmmR0ShModCalcHash.
4381 * @param cbModule The module size.
4382 * @param enmGuestOS The guest OS type.
4383 * @param cRegions The number of regions.
4384 * @param pszModuleName The module name.
4385 * @param pszVersion The module version.
4386 * @param paRegions The region descriptions.
4387 * @param ppGblMod Where to return the new module on success.
4388 */
4389static int gmmR0ShModNewGlobal(PGMM pGMM, uint32_t uHash, uint32_t cbModule, VBOXOSFAMILY enmGuestOS,
4390 uint32_t cRegions, const char *pszModuleName, const char *pszVersion,
4391 struct VMMDEVSHAREDREGIONDESC const *paRegions, PGMMSHAREDMODULE *ppGblMod)
4392{
4393 Log(("gmmR0ShModNewGlobal: %s %s size %#x os %u rgn %u\n", pszModuleName, pszVersion, cbModule, enmGuestOS, cRegions));
4394 if (pGMM->cShareableModules >= GMM_MAX_SHARED_GLOBAL_MODULES)
4395 {
4396 Log(("gmmR0ShModNewGlobal: Too many modules\n"));
4397 return VERR_GMM_TOO_MANY_GLOBAL_MODULES;
4398 }
4399
4400 PGMMSHAREDMODULE pGblMod = (PGMMSHAREDMODULE)RTMemAllocZ(RT_OFFSETOF(GMMSHAREDMODULE, aRegions[cRegions]));
4401 if (!pGblMod)
4402 {
4403 Log(("gmmR0ShModNewGlobal: No memory\n"));
4404 return VERR_NO_MEMORY;
4405 }
4406
4407 pGblMod->Core.Key = uHash;
4408 pGblMod->cbModule = cbModule;
4409 pGblMod->cRegions = cRegions;
4410 pGblMod->cUsers = 1;
4411 pGblMod->enmGuestOS = enmGuestOS;
4412 strcpy(pGblMod->szName, pszModuleName);
4413 strcpy(pGblMod->szVersion, pszVersion);
4414
4415 for (uint32_t i = 0; i < cRegions; i++)
4416 {
4417 Log(("gmmR0ShModNewGlobal: rgn[%u]=%RGvLB%#x\n", i, paRegions[i].GCRegionAddr, paRegions[i].cbRegion));
4418 pGblMod->aRegions[i].off = paRegions[i].GCRegionAddr & PAGE_OFFSET_MASK;
4419 pGblMod->aRegions[i].cb = paRegions[i].cbRegion + pGblMod->aRegions[i].off;
4420 pGblMod->aRegions[i].cb = RT_ALIGN_32(pGblMod->aRegions[i].cb, PAGE_SIZE);
4421 pGblMod->aRegions[i].paidPages = NULL; /* allocated when needed. */
4422 }
4423
4424 bool fInsert = RTAvllU32Insert(&pGMM->pGlobalSharedModuleTree, &pGblMod->Core);
4425 Assert(fInsert); NOREF(fInsert);
4426 pGMM->cShareableModules++;
4427
4428 *ppGblMod = pGblMod;
4429 return VINF_SUCCESS;
4430}
4431
4432
4433/**
4434 * Deletes a global module which is no longer referenced by anyone.
4435 *
4436 * @param pGMM The GMM instance data.
4437 * @param pGblMod The module to delete.
4438 */
4439static void gmmR0ShModDeleteGlobal(PGMM pGMM, PGMMSHAREDMODULE pGblMod)
4440{
4441 Assert(pGblMod->cUsers == 0);
4442 Assert(pGMM->cShareableModules > 0 && pGMM->cShareableModules <= GMM_MAX_SHARED_GLOBAL_MODULES);
4443
4444 void *pvTest = RTAvllU32RemoveNode(&pGMM->pGlobalSharedModuleTree, &pGblMod->Core);
4445 Assert(pvTest == pGblMod); NOREF(pvTest);
4446 pGMM->cShareableModules--;
4447
4448 uint32_t i = pGblMod->cRegions;
4449 while (i-- > 0)
4450 {
4451 if (pGblMod->aRegions[i].paidPages)
4452 {
4453 /* We don't doing anything to the pages as they are handled by the
4454 copy-on-write mechanism in PGM. */
4455 RTMemFree(pGblMod->aRegions[i].paidPages);
4456 pGblMod->aRegions[i].paidPages = NULL;
4457 }
4458 }
4459 RTMemFree(pGblMod);
4460}
4461
4462
4463static int gmmR0ShModNewPerVM(PGVM pGVM, RTGCPTR GCBaseAddr, uint32_t cRegions, const VMMDEVSHAREDREGIONDESC *paRegions,
4464 PGMMSHAREDMODULEPERVM *ppRecVM)
4465{
4466 if (pGVM->gmm.s.Stats.cShareableModules >= GMM_MAX_SHARED_PER_VM_MODULES)
4467 return VERR_GMM_TOO_MANY_PER_VM_MODULES;
4468
4469 PGMMSHAREDMODULEPERVM pRecVM;
4470 pRecVM = (PGMMSHAREDMODULEPERVM)RTMemAllocZ(RT_OFFSETOF(GMMSHAREDMODULEPERVM, aRegionsGCPtrs[cRegions]));
4471 if (!pRecVM)
4472 return VERR_NO_MEMORY;
4473
4474 pRecVM->Core.Key = GCBaseAddr;
4475 for (uint32_t i = 0; i < cRegions; i++)
4476 pRecVM->aRegionsGCPtrs[i] = paRegions[i].GCRegionAddr;
4477
4478 bool fInsert = RTAvlGCPtrInsert(&pGVM->gmm.s.pSharedModuleTree, &pRecVM->Core);
4479 Assert(fInsert); NOREF(fInsert);
4480 pGVM->gmm.s.Stats.cShareableModules++;
4481
4482 *ppRecVM = pRecVM;
4483 return VINF_SUCCESS;
4484}
4485
4486
4487static void gmmR0ShModDeletePerVM(PGMM pGMM, PGVM pGVM, PGMMSHAREDMODULEPERVM pRecVM, bool fRemove)
4488{
4489 /*
4490 * Free the per-VM module.
4491 */
4492 PGMMSHAREDMODULE pGblMod = pRecVM->pGlobalModule;
4493 pRecVM->pGlobalModule = NULL;
4494
4495 if (fRemove)
4496 {
4497 void *pvTest = RTAvlGCPtrRemove(&pGVM->gmm.s.pSharedModuleTree, pRecVM->Core.Key);
4498 Assert(pvTest == &pRecVM->Core); NOREF(pvTest);
4499 }
4500
4501 RTMemFree(pRecVM);
4502
4503 /*
4504 * Release the global module.
4505 * (In the registration bailout case, it might not be.)
4506 */
4507 if (pGblMod)
4508 {
4509 Assert(pGblMod->cUsers > 0);
4510 pGblMod->cUsers--;
4511 if (pGblMod->cUsers == 0)
4512 gmmR0ShModDeleteGlobal(pGMM, pGblMod);
4513 }
4514}
4515
4516#endif /* VBOX_WITH_PAGE_SHARING */
4517
4518/**
4519 * Registers a new shared module for the VM.
4520 *
4521 * @returns VBox status code.
4522 * @param pVM The cross context VM structure.
4523 * @param idCpu The VCPU id.
4524 * @param enmGuestOS The guest OS type.
4525 * @param pszModuleName The module name.
4526 * @param pszVersion The module version.
4527 * @param GCPtrModBase The module base address.
4528 * @param cbModule The module size.
4529 * @param cRegions The mumber of shared region descriptors.
4530 * @param paRegions Pointer to an array of shared region(s).
4531 */
4532GMMR0DECL(int) GMMR0RegisterSharedModule(PVM pVM, VMCPUID idCpu, VBOXOSFAMILY enmGuestOS, char *pszModuleName,
4533 char *pszVersion, RTGCPTR GCPtrModBase, uint32_t cbModule,
4534 uint32_t cRegions, struct VMMDEVSHAREDREGIONDESC const *paRegions)
4535{
4536#ifdef VBOX_WITH_PAGE_SHARING
4537 /*
4538 * Validate input and get the basics.
4539 *
4540 * Note! Turns out the module size does necessarily match the size of the
4541 * regions. (iTunes on XP)
4542 */
4543 PGMM pGMM;
4544 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
4545 PGVM pGVM;
4546 int rc = GVMMR0ByVMAndEMT(pVM, idCpu, &pGVM);
4547 if (RT_FAILURE(rc))
4548 return rc;
4549
4550 if (RT_UNLIKELY(cRegions > VMMDEVSHAREDREGIONDESC_MAX))
4551 return VERR_GMM_TOO_MANY_REGIONS;
4552
4553 if (RT_UNLIKELY(cbModule == 0 || cbModule > _1G))
4554 return VERR_GMM_BAD_SHARED_MODULE_SIZE;
4555
4556 uint32_t cbTotal = 0;
4557 for (uint32_t i = 0; i < cRegions; i++)
4558 {
4559 if (RT_UNLIKELY(paRegions[i].cbRegion == 0 || paRegions[i].cbRegion > _1G))
4560 return VERR_GMM_SHARED_MODULE_BAD_REGIONS_SIZE;
4561
4562 cbTotal += paRegions[i].cbRegion;
4563 if (RT_UNLIKELY(cbTotal > _1G))
4564 return VERR_GMM_SHARED_MODULE_BAD_REGIONS_SIZE;
4565 }
4566
4567 AssertPtrReturn(pszModuleName, VERR_INVALID_POINTER);
4568 if (RT_UNLIKELY(!memchr(pszModuleName, '\0', GMM_SHARED_MODULE_MAX_NAME_STRING)))
4569 return VERR_GMM_MODULE_NAME_TOO_LONG;
4570
4571 AssertPtrReturn(pszVersion, VERR_INVALID_POINTER);
4572 if (RT_UNLIKELY(!memchr(pszVersion, '\0', GMM_SHARED_MODULE_MAX_VERSION_STRING)))
4573 return VERR_GMM_MODULE_NAME_TOO_LONG;
4574
4575 uint32_t const uHash = gmmR0ShModCalcHash(pszModuleName, pszVersion);
4576 Log(("GMMR0RegisterSharedModule %s %s base %RGv size %x hash %x\n", pszModuleName, pszVersion, GCPtrModBase, cbModule, uHash));
4577
4578 /*
4579 * Take the semaphore and do some more validations.
4580 */
4581 gmmR0MutexAcquire(pGMM);
4582 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
4583 {
4584 /*
4585 * Check if this module is already locally registered and register
4586 * it if it isn't. The base address is a unique module identifier
4587 * locally.
4588 */
4589 PGMMSHAREDMODULEPERVM pRecVM = (PGMMSHAREDMODULEPERVM)RTAvlGCPtrGet(&pGVM->gmm.s.pSharedModuleTree, GCPtrModBase);
4590 bool fNewModule = pRecVM == NULL;
4591 if (fNewModule)
4592 {
4593 rc = gmmR0ShModNewPerVM(pGVM, GCPtrModBase, cRegions, paRegions, &pRecVM);
4594 if (RT_SUCCESS(rc))
4595 {
4596 /*
4597 * Find a matching global module, register a new one if needed.
4598 */
4599 PGMMSHAREDMODULE pGblMod = gmmR0ShModFindGlobal(pGMM, uHash, cbModule, enmGuestOS, cRegions,
4600 pszModuleName, pszVersion, paRegions);
4601 if (!pGblMod)
4602 {
4603 Assert(fNewModule);
4604 rc = gmmR0ShModNewGlobal(pGMM, uHash, cbModule, enmGuestOS, cRegions,
4605 pszModuleName, pszVersion, paRegions, &pGblMod);
4606 if (RT_SUCCESS(rc))
4607 {
4608 pRecVM->pGlobalModule = pGblMod; /* (One referenced returned by gmmR0ShModNewGlobal.) */
4609 Log(("GMMR0RegisterSharedModule: new module %s %s\n", pszModuleName, pszVersion));
4610 }
4611 else
4612 gmmR0ShModDeletePerVM(pGMM, pGVM, pRecVM, true /*fRemove*/);
4613 }
4614 else
4615 {
4616 Assert(pGblMod->cUsers > 0 && pGblMod->cUsers < UINT32_MAX / 2);
4617 pGblMod->cUsers++;
4618 pRecVM->pGlobalModule = pGblMod;
4619
4620 Log(("GMMR0RegisterSharedModule: new per vm module %s %s, gbl users %d\n", pszModuleName, pszVersion, pGblMod->cUsers));
4621 }
4622 }
4623 }
4624 else
4625 {
4626 /*
4627 * Attempt to re-register an existing module.
4628 */
4629 PGMMSHAREDMODULE pGblMod = gmmR0ShModFindGlobal(pGMM, uHash, cbModule, enmGuestOS, cRegions,
4630 pszModuleName, pszVersion, paRegions);
4631 if (pRecVM->pGlobalModule == pGblMod)
4632 {
4633 Log(("GMMR0RegisterSharedModule: already registered %s %s, gbl users %d\n", pszModuleName, pszVersion, pGblMod->cUsers));
4634 rc = VINF_GMM_SHARED_MODULE_ALREADY_REGISTERED;
4635 }
4636 else
4637 {
4638 /** @todo may have to unregister+register when this happens in case it's caused
4639 * by VBoxService crashing and being restarted... */
4640 Log(("GMMR0RegisterSharedModule: Address clash!\n"
4641 " incoming at %RGvLB%#x %s %s rgns %u\n"
4642 " existing at %RGvLB%#x %s %s rgns %u\n",
4643 GCPtrModBase, cbModule, pszModuleName, pszVersion, cRegions,
4644 pRecVM->Core.Key, pRecVM->pGlobalModule->cbModule, pRecVM->pGlobalModule->szName,
4645 pRecVM->pGlobalModule->szVersion, pRecVM->pGlobalModule->cRegions));
4646 rc = VERR_GMM_SHARED_MODULE_ADDRESS_CLASH;
4647 }
4648 }
4649 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
4650 }
4651 else
4652 rc = VERR_GMM_IS_NOT_SANE;
4653
4654 gmmR0MutexRelease(pGMM);
4655 return rc;
4656#else
4657
4658 NOREF(pVM); NOREF(idCpu); NOREF(enmGuestOS); NOREF(pszModuleName); NOREF(pszVersion);
4659 NOREF(GCPtrModBase); NOREF(cbModule); NOREF(cRegions); NOREF(paRegions);
4660 return VERR_NOT_IMPLEMENTED;
4661#endif
4662}
4663
4664
4665/**
4666 * VMMR0 request wrapper for GMMR0RegisterSharedModule.
4667 *
4668 * @returns see GMMR0RegisterSharedModule.
4669 * @param pVM The cross context VM structure.
4670 * @param idCpu The VCPU id.
4671 * @param pReq Pointer to the request packet.
4672 */
4673GMMR0DECL(int) GMMR0RegisterSharedModuleReq(PVM pVM, VMCPUID idCpu, PGMMREGISTERSHAREDMODULEREQ pReq)
4674{
4675 /*
4676 * Validate input and pass it on.
4677 */
4678 AssertPtrReturn(pVM, VERR_INVALID_POINTER);
4679 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
4680 AssertMsgReturn(pReq->Hdr.cbReq >= sizeof(*pReq) && pReq->Hdr.cbReq == RT_UOFFSETOF(GMMREGISTERSHAREDMODULEREQ, aRegions[pReq->cRegions]), ("%#x != %#x\n", pReq->Hdr.cbReq, sizeof(*pReq)), VERR_INVALID_PARAMETER);
4681
4682 /* Pass back return code in the request packet to preserve informational codes. (VMMR3CallR0 chokes on them) */
4683 pReq->rc = GMMR0RegisterSharedModule(pVM, idCpu, pReq->enmGuestOS, pReq->szName, pReq->szVersion,
4684 pReq->GCBaseAddr, pReq->cbModule, pReq->cRegions, pReq->aRegions);
4685 return VINF_SUCCESS;
4686}
4687
4688
4689/**
4690 * Unregisters a shared module for the VM
4691 *
4692 * @returns VBox status code.
4693 * @param pVM The cross context VM structure.
4694 * @param idCpu The VCPU id.
4695 * @param pszModuleName The module name.
4696 * @param pszVersion The module version.
4697 * @param GCPtrModBase The module base address.
4698 * @param cbModule The module size.
4699 */
4700GMMR0DECL(int) GMMR0UnregisterSharedModule(PVM pVM, VMCPUID idCpu, char *pszModuleName, char *pszVersion,
4701 RTGCPTR GCPtrModBase, uint32_t cbModule)
4702{
4703#ifdef VBOX_WITH_PAGE_SHARING
4704 /*
4705 * Validate input and get the basics.
4706 */
4707 PGMM pGMM;
4708 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
4709 PGVM pGVM;
4710 int rc = GVMMR0ByVMAndEMT(pVM, idCpu, &pGVM);
4711 if (RT_FAILURE(rc))
4712 return rc;
4713
4714 AssertPtrReturn(pszModuleName, VERR_INVALID_POINTER);
4715 AssertPtrReturn(pszVersion, VERR_INVALID_POINTER);
4716 if (RT_UNLIKELY(!memchr(pszModuleName, '\0', GMM_SHARED_MODULE_MAX_NAME_STRING)))
4717 return VERR_GMM_MODULE_NAME_TOO_LONG;
4718 if (RT_UNLIKELY(!memchr(pszVersion, '\0', GMM_SHARED_MODULE_MAX_VERSION_STRING)))
4719 return VERR_GMM_MODULE_NAME_TOO_LONG;
4720
4721 Log(("GMMR0UnregisterSharedModule %s %s base=%RGv size %x\n", pszModuleName, pszVersion, GCPtrModBase, cbModule));
4722
4723 /*
4724 * Take the semaphore and do some more validations.
4725 */
4726 gmmR0MutexAcquire(pGMM);
4727 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
4728 {
4729 /*
4730 * Locate and remove the specified module.
4731 */
4732 PGMMSHAREDMODULEPERVM pRecVM = (PGMMSHAREDMODULEPERVM)RTAvlGCPtrGet(&pGVM->gmm.s.pSharedModuleTree, GCPtrModBase);
4733 if (pRecVM)
4734 {
4735 /** @todo Do we need to do more validations here, like that the
4736 * name + version + cbModule matches? */
4737 NOREF(cbModule);
4738 Assert(pRecVM->pGlobalModule);
4739 gmmR0ShModDeletePerVM(pGMM, pGVM, pRecVM, true /*fRemove*/);
4740 }
4741 else
4742 rc = VERR_GMM_SHARED_MODULE_NOT_FOUND;
4743
4744 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
4745 }
4746 else
4747 rc = VERR_GMM_IS_NOT_SANE;
4748
4749 gmmR0MutexRelease(pGMM);
4750 return rc;
4751#else
4752
4753 NOREF(pVM); NOREF(idCpu); NOREF(pszModuleName); NOREF(pszVersion); NOREF(GCPtrModBase); NOREF(cbModule);
4754 return VERR_NOT_IMPLEMENTED;
4755#endif
4756}
4757
4758
4759/**
4760 * VMMR0 request wrapper for GMMR0UnregisterSharedModule.
4761 *
4762 * @returns see GMMR0UnregisterSharedModule.
4763 * @param pVM The cross context VM structure.
4764 * @param idCpu The VCPU id.
4765 * @param pReq Pointer to the request packet.
4766 */
4767GMMR0DECL(int) GMMR0UnregisterSharedModuleReq(PVM pVM, VMCPUID idCpu, PGMMUNREGISTERSHAREDMODULEREQ pReq)
4768{
4769 /*
4770 * Validate input and pass it on.
4771 */
4772 AssertPtrReturn(pVM, VERR_INVALID_POINTER);
4773 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
4774 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(*pReq), ("%#x != %#x\n", pReq->Hdr.cbReq, sizeof(*pReq)), VERR_INVALID_PARAMETER);
4775
4776 return GMMR0UnregisterSharedModule(pVM, idCpu, pReq->szName, pReq->szVersion, pReq->GCBaseAddr, pReq->cbModule);
4777}
4778
4779#ifdef VBOX_WITH_PAGE_SHARING
4780
4781/**
4782 * Increase the use count of a shared page, the page is known to exist and be valid and such.
4783 *
4784 * @param pGMM Pointer to the GMM instance.
4785 * @param pGVM Pointer to the GVM instance.
4786 * @param pPage The page structure.
4787 */
4788DECLINLINE(void) gmmR0UseSharedPage(PGMM pGMM, PGVM pGVM, PGMMPAGE pPage)
4789{
4790 Assert(pGMM->cSharedPages > 0);
4791 Assert(pGMM->cAllocatedPages > 0);
4792
4793 pGMM->cDuplicatePages++;
4794
4795 pPage->Shared.cRefs++;
4796 pGVM->gmm.s.Stats.cSharedPages++;
4797 pGVM->gmm.s.Stats.Allocated.cBasePages++;
4798}
4799
4800
4801/**
4802 * Converts a private page to a shared page, the page is known to exist and be valid and such.
4803 *
4804 * @param pGMM Pointer to the GMM instance.
4805 * @param pGVM Pointer to the GVM instance.
4806 * @param HCPhys Host physical address
4807 * @param idPage The Page ID
4808 * @param pPage The page structure.
4809 * @param pPageDesc Shared page descriptor
4810 */
4811DECLINLINE(void) gmmR0ConvertToSharedPage(PGMM pGMM, PGVM pGVM, RTHCPHYS HCPhys, uint32_t idPage, PGMMPAGE pPage,
4812 PGMMSHAREDPAGEDESC pPageDesc)
4813{
4814 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, idPage >> GMM_CHUNKID_SHIFT);
4815 Assert(pChunk);
4816 Assert(pChunk->cFree < GMM_CHUNK_NUM_PAGES);
4817 Assert(GMM_PAGE_IS_PRIVATE(pPage));
4818
4819 pChunk->cPrivate--;
4820 pChunk->cShared++;
4821
4822 pGMM->cSharedPages++;
4823
4824 pGVM->gmm.s.Stats.cSharedPages++;
4825 pGVM->gmm.s.Stats.cPrivatePages--;
4826
4827 /* Modify the page structure. */
4828 pPage->Shared.pfn = (uint32_t)(uint64_t)(HCPhys >> PAGE_SHIFT);
4829 pPage->Shared.cRefs = 1;
4830#ifdef VBOX_STRICT
4831 pPageDesc->u32StrictChecksum = gmmR0StrictPageChecksum(pGMM, pGVM, idPage);
4832 pPage->Shared.u14Checksum = pPageDesc->u32StrictChecksum;
4833#else
4834 NOREF(pPageDesc);
4835 pPage->Shared.u14Checksum = 0;
4836#endif
4837 pPage->Shared.u2State = GMM_PAGE_STATE_SHARED;
4838}
4839
4840
4841static int gmmR0SharedModuleCheckPageFirstTime(PGMM pGMM, PGVM pGVM, PGMMSHAREDMODULE pModule,
4842 unsigned idxRegion, unsigned idxPage,
4843 PGMMSHAREDPAGEDESC pPageDesc, PGMMSHAREDREGIONDESC pGlobalRegion)
4844{
4845 NOREF(pModule);
4846
4847 /* Easy case: just change the internal page type. */
4848 PGMMPAGE pPage = gmmR0GetPage(pGMM, pPageDesc->idPage);
4849 AssertMsgReturn(pPage, ("idPage=%#x (GCPhys=%RGp HCPhys=%RHp idxRegion=%#x idxPage=%#x) #1\n",
4850 pPageDesc->idPage, pPageDesc->GCPhys, pPageDesc->HCPhys, idxRegion, idxPage),
4851 VERR_PGM_PHYS_INVALID_PAGE_ID);
4852 NOREF(idxRegion);
4853
4854 AssertMsg(pPageDesc->GCPhys == (pPage->Private.pfn << 12), ("desc %RGp gmm %RGp\n", pPageDesc->HCPhys, (pPage->Private.pfn << 12)));
4855
4856 gmmR0ConvertToSharedPage(pGMM, pGVM, pPageDesc->HCPhys, pPageDesc->idPage, pPage, pPageDesc);
4857
4858 /* Keep track of these references. */
4859 pGlobalRegion->paidPages[idxPage] = pPageDesc->idPage;
4860
4861 return VINF_SUCCESS;
4862}
4863
4864/**
4865 * Checks specified shared module range for changes
4866 *
4867 * Performs the following tasks:
4868 * - If a shared page is new, then it changes the GMM page type to shared and
4869 * returns it in the pPageDesc descriptor.
4870 * - If a shared page already exists, then it checks if the VM page is
4871 * identical and if so frees the VM page and returns the shared page in
4872 * pPageDesc descriptor.
4873 *
4874 * @remarks ASSUMES the caller has acquired the GMM semaphore!!
4875 *
4876 * @returns VBox status code.
4877 * @param pGVM Pointer to the GVM instance data.
4878 * @param pModule Module description
4879 * @param idxRegion Region index
4880 * @param idxPage Page index
4881 * @param pPageDesc Page descriptor
4882 */
4883GMMR0DECL(int) GMMR0SharedModuleCheckPage(PGVM pGVM, PGMMSHAREDMODULE pModule, uint32_t idxRegion, uint32_t idxPage,
4884 PGMMSHAREDPAGEDESC pPageDesc)
4885{
4886 int rc;
4887 PGMM pGMM;
4888 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
4889 pPageDesc->u32StrictChecksum = 0;
4890
4891 AssertMsgReturn(idxRegion < pModule->cRegions,
4892 ("idxRegion=%#x cRegions=%#x %s %s\n", idxRegion, pModule->cRegions, pModule->szName, pModule->szVersion),
4893 VERR_INVALID_PARAMETER);
4894
4895 uint32_t const cPages = pModule->aRegions[idxRegion].cb >> PAGE_SHIFT;
4896 AssertMsgReturn(idxPage < cPages,
4897 ("idxRegion=%#x cRegions=%#x %s %s\n", idxRegion, pModule->cRegions, pModule->szName, pModule->szVersion),
4898 VERR_INVALID_PARAMETER);
4899
4900 LogFlow(("GMMR0SharedModuleCheckRange %s base %RGv region %d idxPage %d\n", pModule->szName, pModule->Core.Key, idxRegion, idxPage));
4901
4902 /*
4903 * First time; create a page descriptor array.
4904 */
4905 PGMMSHAREDREGIONDESC pGlobalRegion = &pModule->aRegions[idxRegion];
4906 if (!pGlobalRegion->paidPages)
4907 {
4908 Log(("Allocate page descriptor array for %d pages\n", cPages));
4909 pGlobalRegion->paidPages = (uint32_t *)RTMemAlloc(cPages * sizeof(pGlobalRegion->paidPages[0]));
4910 AssertReturn(pGlobalRegion->paidPages, VERR_NO_MEMORY);
4911
4912 /* Invalidate all descriptors. */
4913 uint32_t i = cPages;
4914 while (i-- > 0)
4915 pGlobalRegion->paidPages[i] = NIL_GMM_PAGEID;
4916 }
4917
4918 /*
4919 * We've seen this shared page for the first time?
4920 */
4921 if (pGlobalRegion->paidPages[idxPage] == NIL_GMM_PAGEID)
4922 {
4923 Log(("New shared page guest %RGp host %RHp\n", pPageDesc->GCPhys, pPageDesc->HCPhys));
4924 return gmmR0SharedModuleCheckPageFirstTime(pGMM, pGVM, pModule, idxRegion, idxPage, pPageDesc, pGlobalRegion);
4925 }
4926
4927 /*
4928 * We've seen it before...
4929 */
4930 Log(("Replace existing page guest %RGp host %RHp id %#x -> id %#x\n",
4931 pPageDesc->GCPhys, pPageDesc->HCPhys, pPageDesc->idPage, pGlobalRegion->paidPages[idxPage]));
4932 Assert(pPageDesc->idPage != pGlobalRegion->paidPages[idxPage]);
4933
4934 /*
4935 * Get the shared page source.
4936 */
4937 PGMMPAGE pPage = gmmR0GetPage(pGMM, pGlobalRegion->paidPages[idxPage]);
4938 AssertMsgReturn(pPage, ("idPage=%#x (idxRegion=%#x idxPage=%#x) #2\n", pPageDesc->idPage, idxRegion, idxPage),
4939 VERR_PGM_PHYS_INVALID_PAGE_ID);
4940
4941 if (pPage->Common.u2State != GMM_PAGE_STATE_SHARED)
4942 {
4943 /*
4944 * Page was freed at some point; invalidate this entry.
4945 */
4946 /** @todo this isn't really bullet proof. */
4947 Log(("Old shared page was freed -> create a new one\n"));
4948 pGlobalRegion->paidPages[idxPage] = NIL_GMM_PAGEID;
4949 return gmmR0SharedModuleCheckPageFirstTime(pGMM, pGVM, pModule, idxRegion, idxPage, pPageDesc, pGlobalRegion);
4950 }
4951
4952 Log(("Replace existing page guest host %RHp -> %RHp\n", pPageDesc->HCPhys, ((uint64_t)pPage->Shared.pfn) << PAGE_SHIFT));
4953
4954 /*
4955 * Calculate the virtual address of the local page.
4956 */
4957 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, pPageDesc->idPage >> GMM_CHUNKID_SHIFT);
4958 AssertMsgReturn(pChunk, ("idPage=%#x (idxRegion=%#x idxPage=%#x) #4\n", pPageDesc->idPage, idxRegion, idxPage),
4959 VERR_PGM_PHYS_INVALID_PAGE_ID);
4960
4961 uint8_t *pbChunk;
4962 AssertMsgReturn(gmmR0IsChunkMapped(pGMM, pGVM, pChunk, (PRTR3PTR)&pbChunk),
4963 ("idPage=%#x (idxRegion=%#x idxPage=%#x) #3\n", pPageDesc->idPage, idxRegion, idxPage),
4964 VERR_PGM_PHYS_INVALID_PAGE_ID);
4965 uint8_t *pbLocalPage = pbChunk + ((pPageDesc->idPage & GMM_PAGEID_IDX_MASK) << PAGE_SHIFT);
4966
4967 /*
4968 * Calculate the virtual address of the shared page.
4969 */
4970 pChunk = gmmR0GetChunk(pGMM, pGlobalRegion->paidPages[idxPage] >> GMM_CHUNKID_SHIFT);
4971 Assert(pChunk); /* can't fail as gmmR0GetPage succeeded. */
4972
4973 /*
4974 * Get the virtual address of the physical page; map the chunk into the VM
4975 * process if not already done.
4976 */
4977 if (!gmmR0IsChunkMapped(pGMM, pGVM, pChunk, (PRTR3PTR)&pbChunk))
4978 {
4979 Log(("Map chunk into process!\n"));
4980 rc = gmmR0MapChunk(pGMM, pGVM, pChunk, false /*fRelaxedSem*/, (PRTR3PTR)&pbChunk);
4981 AssertRCReturn(rc, rc);
4982 }
4983 uint8_t *pbSharedPage = pbChunk + ((pGlobalRegion->paidPages[idxPage] & GMM_PAGEID_IDX_MASK) << PAGE_SHIFT);
4984
4985#ifdef VBOX_STRICT
4986 pPageDesc->u32StrictChecksum = RTCrc32(pbSharedPage, PAGE_SIZE);
4987 uint32_t uChecksum = pPageDesc->u32StrictChecksum & UINT32_C(0x00003fff);
4988 AssertMsg(!uChecksum || uChecksum == pPage->Shared.u14Checksum || !pPage->Shared.u14Checksum,
4989 ("%#x vs %#x - idPage=%#x - %s %s\n", uChecksum, pPage->Shared.u14Checksum,
4990 pGlobalRegion->paidPages[idxPage], pModule->szName, pModule->szVersion));
4991#endif
4992
4993 /** @todo write ASMMemComparePage. */
4994 if (memcmp(pbSharedPage, pbLocalPage, PAGE_SIZE))
4995 {
4996 Log(("Unexpected differences found between local and shared page; skip\n"));
4997 /* Signal to the caller that this one hasn't changed. */
4998 pPageDesc->idPage = NIL_GMM_PAGEID;
4999 return VINF_SUCCESS;
5000 }
5001
5002 /*
5003 * Free the old local page.
5004 */
5005 GMMFREEPAGEDESC PageDesc;
5006 PageDesc.idPage = pPageDesc->idPage;
5007 rc = gmmR0FreePages(pGMM, pGVM, 1, &PageDesc, GMMACCOUNT_BASE);
5008 AssertRCReturn(rc, rc);
5009
5010 gmmR0UseSharedPage(pGMM, pGVM, pPage);
5011
5012 /*
5013 * Pass along the new physical address & page id.
5014 */
5015 pPageDesc->HCPhys = ((uint64_t)pPage->Shared.pfn) << PAGE_SHIFT;
5016 pPageDesc->idPage = pGlobalRegion->paidPages[idxPage];
5017
5018 return VINF_SUCCESS;
5019}
5020
5021
5022/**
5023 * RTAvlGCPtrDestroy callback.
5024 *
5025 * @returns 0 or VERR_GMM_INSTANCE.
5026 * @param pNode The node to destroy.
5027 * @param pvArgs Pointer to an argument packet.
5028 */
5029static DECLCALLBACK(int) gmmR0CleanupSharedModule(PAVLGCPTRNODECORE pNode, void *pvArgs)
5030{
5031 gmmR0ShModDeletePerVM(((GMMR0SHMODPERVMDTORARGS *)pvArgs)->pGMM,
5032 ((GMMR0SHMODPERVMDTORARGS *)pvArgs)->pGVM,
5033 (PGMMSHAREDMODULEPERVM)pNode,
5034 false /*fRemove*/);
5035 return VINF_SUCCESS;
5036}
5037
5038
5039/**
5040 * Used by GMMR0CleanupVM to clean up shared modules.
5041 *
5042 * This is called without taking the GMM lock so that it can be yielded as
5043 * needed here.
5044 *
5045 * @param pGMM The GMM handle.
5046 * @param pGVM The global VM handle.
5047 */
5048static void gmmR0SharedModuleCleanup(PGMM pGMM, PGVM pGVM)
5049{
5050 gmmR0MutexAcquire(pGMM);
5051 GMM_CHECK_SANITY_UPON_ENTERING(pGMM);
5052
5053 GMMR0SHMODPERVMDTORARGS Args;
5054 Args.pGVM = pGVM;
5055 Args.pGMM = pGMM;
5056 RTAvlGCPtrDestroy(&pGVM->gmm.s.pSharedModuleTree, gmmR0CleanupSharedModule, &Args);
5057
5058 AssertMsg(pGVM->gmm.s.Stats.cShareableModules == 0, ("%d\n", pGVM->gmm.s.Stats.cShareableModules));
5059 pGVM->gmm.s.Stats.cShareableModules = 0;
5060
5061 gmmR0MutexRelease(pGMM);
5062}
5063
5064#endif /* VBOX_WITH_PAGE_SHARING */
5065
5066/**
5067 * Removes all shared modules for the specified VM
5068 *
5069 * @returns VBox status code.
5070 * @param pVM The cross context VM structure.
5071 * @param idCpu The VCPU id.
5072 */
5073GMMR0DECL(int) GMMR0ResetSharedModules(PVM pVM, VMCPUID idCpu)
5074{
5075#ifdef VBOX_WITH_PAGE_SHARING
5076 /*
5077 * Validate input and get the basics.
5078 */
5079 PGMM pGMM;
5080 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
5081 PGVM pGVM;
5082 int rc = GVMMR0ByVMAndEMT(pVM, idCpu, &pGVM);
5083 if (RT_FAILURE(rc))
5084 return rc;
5085
5086 /*
5087 * Take the semaphore and do some more validations.
5088 */
5089 gmmR0MutexAcquire(pGMM);
5090 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
5091 {
5092 Log(("GMMR0ResetSharedModules\n"));
5093 GMMR0SHMODPERVMDTORARGS Args;
5094 Args.pGVM = pGVM;
5095 Args.pGMM = pGMM;
5096 RTAvlGCPtrDestroy(&pGVM->gmm.s.pSharedModuleTree, gmmR0CleanupSharedModule, &Args);
5097 pGVM->gmm.s.Stats.cShareableModules = 0;
5098
5099 rc = VINF_SUCCESS;
5100 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
5101 }
5102 else
5103 rc = VERR_GMM_IS_NOT_SANE;
5104
5105 gmmR0MutexRelease(pGMM);
5106 return rc;
5107#else
5108 NOREF(pVM); NOREF(idCpu);
5109 return VERR_NOT_IMPLEMENTED;
5110#endif
5111}
5112
5113#ifdef VBOX_WITH_PAGE_SHARING
5114
5115/**
5116 * Tree enumeration callback for checking a shared module.
5117 */
5118static DECLCALLBACK(int) gmmR0CheckSharedModule(PAVLGCPTRNODECORE pNode, void *pvUser)
5119{
5120 GMMCHECKSHAREDMODULEINFO *pArgs = (GMMCHECKSHAREDMODULEINFO*)pvUser;
5121 PGMMSHAREDMODULEPERVM pRecVM = (PGMMSHAREDMODULEPERVM)pNode;
5122 PGMMSHAREDMODULE pGblMod = pRecVM->pGlobalModule;
5123
5124 Log(("gmmR0CheckSharedModule: check %s %s base=%RGv size=%x\n",
5125 pGblMod->szName, pGblMod->szVersion, pGblMod->Core.Key, pGblMod->cbModule));
5126
5127 int rc = PGMR0SharedModuleCheck(pArgs->pGVM->pVM, pArgs->pGVM, pArgs->idCpu, pGblMod, pRecVM->aRegionsGCPtrs);
5128 if (RT_FAILURE(rc))
5129 return rc;
5130 return VINF_SUCCESS;
5131}
5132
5133#endif /* VBOX_WITH_PAGE_SHARING */
5134#ifdef DEBUG_sandervl
5135
5136/**
5137 * Setup for a GMMR0CheckSharedModules call (to allow log flush jumps back to ring 3)
5138 *
5139 * @returns VBox status code.
5140 * @param pVM The cross context VM structure.
5141 */
5142GMMR0DECL(int) GMMR0CheckSharedModulesStart(PVM pVM)
5143{
5144 /*
5145 * Validate input and get the basics.
5146 */
5147 PGMM pGMM;
5148 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
5149
5150 /*
5151 * Take the semaphore and do some more validations.
5152 */
5153 gmmR0MutexAcquire(pGMM);
5154 if (!GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
5155 rc = VERR_GMM_IS_NOT_SANE;
5156 else
5157 rc = VINF_SUCCESS;
5158
5159 return rc;
5160}
5161
5162/**
5163 * Clean up after a GMMR0CheckSharedModules call (to allow log flush jumps back to ring 3)
5164 *
5165 * @returns VBox status code.
5166 * @param pVM The cross context VM structure.
5167 */
5168GMMR0DECL(int) GMMR0CheckSharedModulesEnd(PVM pVM)
5169{
5170 /*
5171 * Validate input and get the basics.
5172 */
5173 PGMM pGMM;
5174 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
5175
5176 gmmR0MutexRelease(pGMM);
5177 return VINF_SUCCESS;
5178}
5179
5180#endif /* DEBUG_sandervl */
5181
5182/**
5183 * Check all shared modules for the specified VM.
5184 *
5185 * @returns VBox status code.
5186 * @param pVM The cross context VM structure.
5187 * @param pVCpu The cross context virtual CPU structure.
5188 */
5189GMMR0DECL(int) GMMR0CheckSharedModules(PVM pVM, PVMCPU pVCpu)
5190{
5191#ifdef VBOX_WITH_PAGE_SHARING
5192 /*
5193 * Validate input and get the basics.
5194 */
5195 PGMM pGMM;
5196 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
5197 PGVM pGVM;
5198 int rc = GVMMR0ByVMAndEMT(pVM, pVCpu->idCpu, &pGVM);
5199 if (RT_FAILURE(rc))
5200 return rc;
5201
5202# ifndef DEBUG_sandervl
5203 /*
5204 * Take the semaphore and do some more validations.
5205 */
5206 gmmR0MutexAcquire(pGMM);
5207# endif
5208 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
5209 {
5210 /*
5211 * Walk the tree, checking each module.
5212 */
5213 Log(("GMMR0CheckSharedModules\n"));
5214
5215 GMMCHECKSHAREDMODULEINFO Args;
5216 Args.pGVM = pGVM;
5217 Args.idCpu = pVCpu->idCpu;
5218 rc = RTAvlGCPtrDoWithAll(&pGVM->gmm.s.pSharedModuleTree, true /* fFromLeft */, gmmR0CheckSharedModule, &Args);
5219
5220 Log(("GMMR0CheckSharedModules done (rc=%Rrc)!\n", rc));
5221 GMM_CHECK_SANITY_UPON_LEAVING(pGMM);
5222 }
5223 else
5224 rc = VERR_GMM_IS_NOT_SANE;
5225
5226# ifndef DEBUG_sandervl
5227 gmmR0MutexRelease(pGMM);
5228# endif
5229 return rc;
5230#else
5231 NOREF(pVM); NOREF(pVCpu);
5232 return VERR_NOT_IMPLEMENTED;
5233#endif
5234}
5235
5236#if defined(VBOX_STRICT) && HC_ARCH_BITS == 64
5237
5238/**
5239 * RTAvlU32DoWithAll callback.
5240 *
5241 * @returns 0
5242 * @param pNode The node to search.
5243 * @param pvUser Pointer to the input argument packet.
5244 */
5245static DECLCALLBACK(int) gmmR0FindDupPageInChunk(PAVLU32NODECORE pNode, void *pvUser)
5246{
5247 PGMMCHUNK pChunk = (PGMMCHUNK)pNode;
5248 GMMFINDDUPPAGEINFO *pArgs = (GMMFINDDUPPAGEINFO *)pvUser;
5249 PGVM pGVM = pArgs->pGVM;
5250 PGMM pGMM = pArgs->pGMM;
5251 uint8_t *pbChunk;
5252
5253 /* Only take chunks not mapped into this VM process; not entirely correct. */
5254 if (!gmmR0IsChunkMapped(pGMM, pGVM, pChunk, (PRTR3PTR)&pbChunk))
5255 {
5256 int rc = gmmR0MapChunk(pGMM, pGVM, pChunk, false /*fRelaxedSem*/, (PRTR3PTR)&pbChunk);
5257 if (RT_SUCCESS(rc))
5258 {
5259 /*
5260 * Look for duplicate pages
5261 */
5262 unsigned iPage = (GMM_CHUNK_SIZE >> PAGE_SHIFT);
5263 while (iPage-- > 0)
5264 {
5265 if (GMM_PAGE_IS_PRIVATE(&pChunk->aPages[iPage]))
5266 {
5267 uint8_t *pbDestPage = pbChunk + (iPage << PAGE_SHIFT);
5268
5269 if (!memcmp(pArgs->pSourcePage, pbDestPage, PAGE_SIZE))
5270 {
5271 pArgs->fFoundDuplicate = true;
5272 break;
5273 }
5274 }
5275 }
5276 gmmR0UnmapChunk(pGMM, pGVM, pChunk, false /*fRelaxedSem*/);
5277 }
5278 }
5279 return pArgs->fFoundDuplicate; /* (stops search if true) */
5280}
5281
5282
5283/**
5284 * Find a duplicate of the specified page in other active VMs
5285 *
5286 * @returns VBox status code.
5287 * @param pVM The cross context VM structure.
5288 * @param pReq Pointer to the request packet.
5289 */
5290GMMR0DECL(int) GMMR0FindDuplicatePageReq(PVM pVM, PGMMFINDDUPLICATEPAGEREQ pReq)
5291{
5292 /*
5293 * Validate input and pass it on.
5294 */
5295 AssertPtrReturn(pVM, VERR_INVALID_POINTER);
5296 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
5297 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(*pReq), ("%#x != %#x\n", pReq->Hdr.cbReq, sizeof(*pReq)), VERR_INVALID_PARAMETER);
5298
5299 PGMM pGMM;
5300 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
5301
5302 PGVM pGVM;
5303 int rc = GVMMR0ByVM(pVM, &pGVM);
5304 if (RT_FAILURE(rc))
5305 return rc;
5306
5307 /*
5308 * Take the semaphore and do some more validations.
5309 */
5310 rc = gmmR0MutexAcquire(pGMM);
5311 if (GMM_CHECK_SANITY_UPON_ENTERING(pGMM))
5312 {
5313 uint8_t *pbChunk;
5314 PGMMCHUNK pChunk = gmmR0GetChunk(pGMM, pReq->idPage >> GMM_CHUNKID_SHIFT);
5315 if (pChunk)
5316 {
5317 if (gmmR0IsChunkMapped(pGMM, pGVM, pChunk, (PRTR3PTR)&pbChunk))
5318 {
5319 uint8_t *pbSourcePage = pbChunk + ((pReq->idPage & GMM_PAGEID_IDX_MASK) << PAGE_SHIFT);
5320 PGMMPAGE pPage = gmmR0GetPage(pGMM, pReq->idPage);
5321 if (pPage)
5322 {
5323 GMMFINDDUPPAGEINFO Args;
5324 Args.pGVM = pGVM;
5325 Args.pGMM = pGMM;
5326 Args.pSourcePage = pbSourcePage;
5327 Args.fFoundDuplicate = false;
5328 RTAvlU32DoWithAll(&pGMM->pChunks, true /* fFromLeft */, gmmR0FindDupPageInChunk, &Args);
5329
5330 pReq->fDuplicate = Args.fFoundDuplicate;
5331 }
5332 else
5333 {
5334 AssertFailed();
5335 rc = VERR_PGM_PHYS_INVALID_PAGE_ID;
5336 }
5337 }
5338 else
5339 AssertFailed();
5340 }
5341 else
5342 AssertFailed();
5343 }
5344 else
5345 rc = VERR_GMM_IS_NOT_SANE;
5346
5347 gmmR0MutexRelease(pGMM);
5348 return rc;
5349}
5350
5351#endif /* VBOX_STRICT && HC_ARCH_BITS == 64 */
5352
5353
5354/**
5355 * Retrieves the GMM statistics visible to the caller.
5356 *
5357 * @returns VBox status code.
5358 *
5359 * @param pStats Where to put the statistics.
5360 * @param pSession The current session.
5361 * @param pVM The VM to obtain statistics for. Optional.
5362 */
5363GMMR0DECL(int) GMMR0QueryStatistics(PGMMSTATS pStats, PSUPDRVSESSION pSession, PVM pVM)
5364{
5365 LogFlow(("GVMMR0QueryStatistics: pStats=%p pSession=%p pVM=%p\n", pStats, pSession, pVM));
5366
5367 /*
5368 * Validate input.
5369 */
5370 AssertPtrReturn(pSession, VERR_INVALID_POINTER);
5371 AssertPtrReturn(pStats, VERR_INVALID_POINTER);
5372 pStats->cMaxPages = 0; /* (crash before taking the mutex...) */
5373
5374 PGMM pGMM;
5375 GMM_GET_VALID_INSTANCE(pGMM, VERR_GMM_INSTANCE);
5376
5377 /*
5378 * Resolve the VM handle, if not NULL, and lock the GMM.
5379 */
5380 int rc;
5381 PGVM pGVM;
5382 if (pVM)
5383 {
5384 rc = GVMMR0ByVM(pVM, &pGVM);
5385 if (RT_FAILURE(rc))
5386 return rc;
5387 }
5388 else
5389 pGVM = NULL;
5390
5391 rc = gmmR0MutexAcquire(pGMM);
5392 if (RT_FAILURE(rc))
5393 return rc;
5394
5395 /*
5396 * Copy out the GMM statistics.
5397 */
5398 pStats->cMaxPages = pGMM->cMaxPages;
5399 pStats->cReservedPages = pGMM->cReservedPages;
5400 pStats->cOverCommittedPages = pGMM->cOverCommittedPages;
5401 pStats->cAllocatedPages = pGMM->cAllocatedPages;
5402 pStats->cSharedPages = pGMM->cSharedPages;
5403 pStats->cDuplicatePages = pGMM->cDuplicatePages;
5404 pStats->cLeftBehindSharedPages = pGMM->cLeftBehindSharedPages;
5405 pStats->cBalloonedPages = pGMM->cBalloonedPages;
5406 pStats->cChunks = pGMM->cChunks;
5407 pStats->cFreedChunks = pGMM->cFreedChunks;
5408 pStats->cShareableModules = pGMM->cShareableModules;
5409 RT_ZERO(pStats->au64Reserved);
5410
5411 /*
5412 * Copy out the VM statistics.
5413 */
5414 if (pGVM)
5415 pStats->VMStats = pGVM->gmm.s.Stats;
5416 else
5417 RT_ZERO(pStats->VMStats);
5418
5419 gmmR0MutexRelease(pGMM);
5420 return rc;
5421}
5422
5423
5424/**
5425 * VMMR0 request wrapper for GMMR0QueryStatistics.
5426 *
5427 * @returns see GMMR0QueryStatistics.
5428 * @param pVM The cross context VM structure. Optional.
5429 * @param pReq Pointer to the request packet.
5430 */
5431GMMR0DECL(int) GMMR0QueryStatisticsReq(PVM pVM, PGMMQUERYSTATISTICSSREQ pReq)
5432{
5433 /*
5434 * Validate input and pass it on.
5435 */
5436 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
5437 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(*pReq), ("%#x != %#x\n", pReq->Hdr.cbReq, sizeof(*pReq)), VERR_INVALID_PARAMETER);
5438
5439 return GMMR0QueryStatistics(&pReq->Stats, pReq->pSession, pVM);
5440}
5441
5442
5443/**
5444 * Resets the specified GMM statistics.
5445 *
5446 * @returns VBox status code.
5447 *
5448 * @param pStats Which statistics to reset, that is, non-zero fields
5449 * indicates which to reset.
5450 * @param pSession The current session.
5451 * @param pVM The VM to reset statistics for. Optional.
5452 */
5453GMMR0DECL(int) GMMR0ResetStatistics(PCGMMSTATS pStats, PSUPDRVSESSION pSession, PVM pVM)
5454{
5455 NOREF(pStats); NOREF(pSession); NOREF(pVM);
5456 /* Currently nothing we can reset at the moment. */
5457 return VINF_SUCCESS;
5458}
5459
5460
5461/**
5462 * VMMR0 request wrapper for GMMR0ResetStatistics.
5463 *
5464 * @returns see GMMR0ResetStatistics.
5465 * @param pVM The cross context VM structure. Optional.
5466 * @param pReq Pointer to the request packet.
5467 */
5468GMMR0DECL(int) GMMR0ResetStatisticsReq(PVM pVM, PGMMRESETSTATISTICSSREQ pReq)
5469{
5470 /*
5471 * Validate input and pass it on.
5472 */
5473 AssertPtrReturn(pReq, VERR_INVALID_POINTER);
5474 AssertMsgReturn(pReq->Hdr.cbReq == sizeof(*pReq), ("%#x != %#x\n", pReq->Hdr.cbReq, sizeof(*pReq)), VERR_INVALID_PARAMETER);
5475
5476 return GMMR0ResetStatistics(&pReq->Stats, pReq->pSession, pVM);
5477}
5478
Note: See TracBrowser for help on using the repository browser.

© 2024 Oracle Support Privacy / Do Not Sell My Info Terms of Use Trademark Policy Automated Access Etiquette