PGM.cpp@ 36701

Last change on this file since 36701 was 36627, checked in by vboxsync, 14 years ago
Fix regression introduced with r70774. Change the RC structure of the mapping cache instead of the R0 one
Property svn:eol-style set to `native` Property svn:keywords set to `Id`
File size: 199.2 KB

Line
1	/* $Id: PGM.cpp 36627 2011-04-08 15:38:47Z vboxsync $ */
2	/** @file
3	* PGM - Page Manager and Monitor. (Mixing stuff here, not good?)
4	*/
5
6	/*
7	* Copyright (C) 2006-2010 Oracle Corporation
8	*
9	* This file is part of VirtualBox Open Source Edition (OSE), as
10	* available from http://www.virtualbox.org. This file is free software;
11	* you can redistribute it and/or modify it under the terms of the GNU
12	* General Public License (GPL) as published by the Free Software
13	* Foundation, in version 2 as it comes in the "COPYING" file of the
14	* VirtualBox OSE distribution. VirtualBox OSE is distributed in the
15	* hope that it will be useful, but WITHOUT ANY WARRANTY of any kind.
16	*/
17
18
19	/** @page pg_pgm PGM - The Page Manager and Monitor
20	*
21	* @see grp_pgm,
22	* @ref pg_pgm_pool,
23	* @ref pg_pgm_phys.
24	*
25	*
26	* @section sec_pgm_modes Paging Modes
27	*
28	* There are three memory contexts: Host Context (HC), Guest Context (GC)
29	* and intermediate context. When talking about paging HC can also be referred to
30	* as "host paging", and GC referred to as "shadow paging".
31	*
32	* We define three basic paging modes: 32-bit, PAE and AMD64. The host paging mode
33	* is defined by the host operating system. The mode used in the shadow paging mode
34	* depends on the host paging mode and what the mode the guest is currently in. The
35	* following relation between the two is defined:
36	*
37	* @verbatim
38	Host > 32-bit \| PAE \| AMD64 \|
39	Guest \| \| \| \|
40	==v================================
41	32-bit 32-bit PAE PAE
42	-------\|--------\|--------\|--------\|
43	PAE PAE PAE PAE
44	-------\|--------\|--------\|--------\|
45	AMD64 AMD64 AMD64 AMD64
46	-------\|--------\|--------\|--------\| @endverbatim
47	*
48	* All configuration except those in the diagonal (upper left) are expected to
49	* require special effort from the switcher (i.e. a bit slower).
50	*
51	*
52	*
53	*
54	* @section sec_pgm_shw The Shadow Memory Context
55	*
56	*
57	* [..]
58	*
59	* Because of guest context mappings requires PDPT and PML4 entries to allow
60	* writing on AMD64, the two upper levels will have fixed flags whatever the
61	* guest is thinking of using there. So, when shadowing the PD level we will
62	* calculate the effective flags of PD and all the higher levels. In legacy
63	* PAE mode this only applies to the PWT and PCD bits (the rest are
64	* ignored/reserved/MBZ). We will ignore those bits for the present.
65	*
66	*
67	*
68	* @section sec_pgm_int The Intermediate Memory Context
69	*
70	* The world switch goes thru an intermediate memory context which purpose it is
71	* to provide different mappings of the switcher code. All guest mappings are also
72	* present in this context.
73	*
74	* The switcher code is mapped at the same location as on the host, at an
75	* identity mapped location (physical equals virtual address), and at the
76	* hypervisor location. The identity mapped location is for when the world
77	* switches that involves disabling paging.
78	*
79	* PGM maintain page tables for 32-bit, PAE and AMD64 paging modes. This
80	* simplifies switching guest CPU mode and consistency at the cost of more
81	* code to do the work. All memory use for those page tables is located below
82	* 4GB (this includes page tables for guest context mappings).
83	*
84	*
85	* @subsection subsec_pgm_int_gc Guest Context Mappings
86	*
87	* During assignment and relocation of a guest context mapping the intermediate
88	* memory context is used to verify the new location.
89	*
90	* Guest context mappings are currently restricted to below 4GB, for reasons
91	* of simplicity. This may change when we implement AMD64 support.
92	*
93	*
94	*
95	*
96	* @section sec_pgm_misc Misc
97	*
98	* @subsection subsec_pgm_misc_diff Differences Between Legacy PAE and Long Mode PAE
99	*
100	* The differences between legacy PAE and long mode PAE are:
101	* -# PDPE bits 1, 2, 5 and 6 are defined differently. In leagcy mode they are
102	* all marked down as must-be-zero, while in long mode 1, 2 and 5 have the
103	* usual meanings while 6 is ignored (AMD). This means that upon switching to
104	* legacy PAE mode we'll have to clear these bits and when going to long mode
105	* they must be set. This applies to both intermediate and shadow contexts,
106	* however we don't need to do it for the intermediate one since we're
107	* executing with CR0.WP at that time.
108	* -# CR3 allows a 32-byte aligned address in legacy mode, while in long mode
109	* a page aligned one is required.
110	*
111	*
112	* @section sec_pgm_handlers Access Handlers
113	*
114	* Placeholder.
115	*
116	*
117	* @subsection sec_pgm_handlers_virt Virtual Access Handlers
118	*
119	* Placeholder.
120	*
121	*
122	* @subsection sec_pgm_handlers_virt Virtual Access Handlers
123	*
124	* We currently implement three types of virtual access handlers: ALL, WRITE
125	* and HYPERVISOR (WRITE). See PGMVIRTHANDLERTYPE for some more details.
126	*
127	* The HYPERVISOR access handlers is kept in a separate tree since it doesn't apply
128	* to physical pages (PGMTREES::HyperVirtHandlers) and only needs to be consulted in
129	* a special \#PF case. The ALL and WRITE are in the PGMTREES::VirtHandlers tree, the
130	* rest of this section is going to be about these handlers.
131	*
132	* We'll go thru the life cycle of a handler and try make sense of it all, don't know
133	* how successful this is gonna be...
134	*
135	* 1. A handler is registered thru the PGMR3HandlerVirtualRegister and
136	* PGMHandlerVirtualRegisterEx APIs. We check for conflicting virtual handlers
137	* and create a new node that is inserted into the AVL tree (range key). Then
138	* a full PGM resync is flagged (clear pool, sync cr3, update virtual bit of PGMPAGE).
139	*
140	* 2. The following PGMSyncCR3/SyncCR3 operation will first make invoke HandlerVirtualUpdate.
141	*
142	* 2a. HandlerVirtualUpdate will will lookup all the pages covered by virtual handlers
143	* via the current guest CR3 and update the physical page -> virtual handler
144	* translation. Needless to say, this doesn't exactly scale very well. If any changes
145	* are detected, it will flag a virtual bit update just like we did on registration.
146	* PGMPHYS pages with changes will have their virtual handler state reset to NONE.
147	*
148	* 2b. The virtual bit update process will iterate all the pages covered by all the
149	* virtual handlers and update the PGMPAGE virtual handler state to the max of all
150	* virtual handlers on that page.
151	*
152	* 2c. Back in SyncCR3 we will now flush the entire shadow page cache to make sure
153	* we don't miss any alias mappings of the monitored pages.
154	*
155	* 2d. SyncCR3 will then proceed with syncing the CR3 table.
156	*
157	* 3. \#PF(np,read) on a page in the range. This will cause it to be synced
158	* read-only and resumed if it's a WRITE handler. If it's an ALL handler we
159	* will call the handlers like in the next step. If the physical mapping has
160	* changed we will - some time in the future - perform a handler callback
161	* (optional) and update the physical -> virtual handler cache.
162	*
163	* 4. \#PF(,write) on a page in the range. This will cause the handler to
164	* be invoked.
165	*
166	* 5. The guest invalidates the page and changes the physical backing or
167	* unmaps it. This should cause the invalidation callback to be invoked
168	* (it might not yet be 100% perfect). Exactly what happens next... is
169	* this where we mess up and end up out of sync for a while?
170	*
171	* 6. The handler is deregistered by the client via PGMHandlerVirtualDeregister.
172	* We will then set all PGMPAGEs in the physical -> virtual handler cache for
173	* this handler to NONE and trigger a full PGM resync (basically the same
174	* as int step 1). Which means 2 is executed again.
175	*
176	*
177	* @subsubsection sub_sec_pgm_handler_virt_todo TODOs
178	*
179	* There is a bunch of things that needs to be done to make the virtual handlers
180	* work 100% correctly and work more efficiently.
181	*
182	* The first bit hasn't been implemented yet because it's going to slow the
183	* whole mess down even more, and besides it seems to be working reliably for
184	* our current uses. OTOH, some of the optimizations might end up more or less
185	* implementing the missing bits, so we'll see.
186	*
187	* On the optimization side, the first thing to do is to try avoid unnecessary
188	* cache flushing. Then try team up with the shadowing code to track changes
189	* in mappings by means of access to them (shadow in), updates to shadows pages,
190	* invlpg, and shadow PT discarding (perhaps).
191	*
192	* Some idea that have popped up for optimization for current and new features:
193	* - bitmap indicating where there are virtual handlers installed.
194	* (4KB => 220 pages, page 212 => covers 32-bit address space 1:1!)
195	* - Further optimize this by min/max (needs min/max avl getters).
196	* - Shadow page table entry bit (if any left)?
197	*
198	*/
199
200
201	/** @page pg_pgm_phys PGM Physical Guest Memory Management
202	*
203	*
204	* Objectives:
205	* - Guest RAM over-commitment using memory ballooning,
206	* zero pages and general page sharing.
207	* - Moving or mirroring a VM onto a different physical machine.
208	*
209	*
210	* @subsection subsec_pgmPhys_Definitions Definitions
211	*
212	* Allocation chunk - A RTR0MemObjAllocPhysNC object and the tracking
213	* machinery associated with it.
214	*
215	*
216	*
217	*
218	* @subsection subsec_pgmPhys_AllocPage Allocating a page.
219	*
220	* Initially we map all guest memory to the (per VM) zero page, which
221	* means that none of the read functions will cause pages to be allocated.
222	*
223	* Exception, access bit in page tables that have been shared. This must
224	* be handled, but we must also make sure PGMGst*Modify doesn't make
225	* unnecessary modifications.
226	*
227	* Allocation points:
228	* - PGMPhysSimpleWriteGCPhys and PGMPhysWrite.
229	* - Replacing a zero page mapping at \#PF.
230	* - Replacing a shared page mapping at \#PF.
231	* - ROM registration (currently MMR3RomRegister).
232	* - VM restore (pgmR3Load).
233	*
234	* For the first three it would make sense to keep a few pages handy
235	* until we've reached the max memory commitment for the VM.
236	*
237	* For the ROM registration, we know exactly how many pages we need
238	* and will request these from ring-0. For restore, we will save
239	* the number of non-zero pages in the saved state and allocate
240	* them up front. This would allow the ring-0 component to refuse
241	* the request if the isn't sufficient memory available for VM use.
242	*
243	* Btw. for both ROM and restore allocations we won't be requiring
244	* zeroed pages as they are going to be filled instantly.
245	*
246	*
247	* @subsection subsec_pgmPhys_FreePage Freeing a page
248	*
249	* There are a few points where a page can be freed:
250	* - After being replaced by the zero page.
251	* - After being replaced by a shared page.
252	* - After being ballooned by the guest additions.
253	* - At reset.
254	* - At restore.
255	*
256	* When freeing one or more pages they will be returned to the ring-0
257	* component and replaced by the zero page.
258	*
259	* The reasoning for clearing out all the pages on reset is that it will
260	* return us to the exact same state as on power on, and may thereby help
261	* us reduce the memory load on the system. Further it might have a
262	* (temporary) positive influence on memory fragmentation (@see subsec_pgmPhys_Fragmentation).
263	*
264	* On restore, as mention under the allocation topic, pages should be
265	* freed / allocated depending on how many is actually required by the
266	* new VM state. The simplest approach is to do like on reset, and free
267	* all non-ROM pages and then allocate what we need.
268	*
269	* A measure to prevent some fragmentation, would be to let each allocation
270	* chunk have some affinity towards the VM having allocated the most pages
271	* from it. Also, try make sure to allocate from allocation chunks that
272	* are almost full. Admittedly, both these measures might work counter to
273	* our intentions and its probably not worth putting a lot of effort,
274	* cpu time or memory into this.
275	*
276	*
277	* @subsection subsec_pgmPhys_SharePage Sharing a page
278	*
279	* The basic idea is that there there will be a idle priority kernel
280	* thread walking the non-shared VM pages hashing them and looking for
281	* pages with the same checksum. If such pages are found, it will compare
282	* them byte-by-byte to see if they actually are identical. If found to be
283	* identical it will allocate a shared page, copy the content, check that
284	* the page didn't change while doing this, and finally request both the
285	* VMs to use the shared page instead. If the page is all zeros (special
286	* checksum and byte-by-byte check) it will request the VM that owns it
287	* to replace it with the zero page.
288	*
289	* To make this efficient, we will have to make sure not to try share a page
290	* that will change its contents soon. This part requires the most work.
291	* A simple idea would be to request the VM to write monitor the page for
292	* a while to make sure it isn't modified any time soon. Also, it may
293	* make sense to skip pages that are being write monitored since this
294	* information is readily available to the thread if it works on the
295	* per-VM guest memory structures (presently called PGMRAMRANGE).
296	*
297	*
298	* @subsection subsec_pgmPhys_Fragmentation Fragmentation Concerns and Counter Measures
299	*
300	* The pages are organized in allocation chunks in ring-0, this is a necessity
301	* if we wish to have an OS agnostic approach to this whole thing. (On Linux we
302	* could easily work on a page-by-page basis if we liked. Whether this is possible
303	* or efficient on NT I don't quite know.) Fragmentation within these chunks may
304	* become a problem as part of the idea here is that we wish to return memory to
305	* the host system.
306	*
307	* For instance, starting two VMs at the same time, they will both allocate the
308	* guest memory on-demand and if permitted their page allocations will be
309	* intermixed. Shut down one of the two VMs and it will be difficult to return
310	* any memory to the host system because the page allocation for the two VMs are
311	* mixed up in the same allocation chunks.
312	*
313	* To further complicate matters, when pages are freed because they have been
314	* ballooned or become shared/zero the whole idea is that the page is supposed
315	* to be reused by another VM or returned to the host system. This will cause
316	* allocation chunks to contain pages belonging to different VMs and prevent
317	* returning memory to the host when one of those VM shuts down.
318	*
319	* The only way to really deal with this problem is to move pages. This can
320	* either be done at VM shutdown and or by the idle priority worker thread
321	* that will be responsible for finding sharable/zero pages. The mechanisms
322	* involved for coercing a VM to move a page (or to do it for it) will be
323	* the same as when telling it to share/zero a page.
324	*
325	*
326	* @subsection subsec_pgmPhys_Tracking Tracking Structures And Their Cost
327	*
328	* There's a difficult balance between keeping the per-page tracking structures
329	* (global and guest page) easy to use and keeping them from eating too much
330	* memory. We have limited virtual memory resources available when operating in
331	* 32-bit kernel space (on 64-bit there'll it's quite a different story). The
332	* tracking structures will be attempted designed such that we can deal with up
333	* to 32GB of memory on a 32-bit system and essentially unlimited on 64-bit ones.
334	*
335	*
336	* @subsubsection subsubsec_pgmPhys_Tracking_Kernel Kernel Space
337	*
338	* @see pg_GMM
339	*
340	* @subsubsection subsubsec_pgmPhys_Tracking_PerVM Per-VM
341	*
342	* Fixed info is the physical address of the page (HCPhys) and the page id
343	* (described above). Theoretically we'll need 48(-12) bits for the HCPhys part.
344	* Today we've restricting ourselves to 40(-12) bits because this is the current
345	* restrictions of all AMD64 implementations (I think Barcelona will up this
346	* to 48(-12) bits, not that it really matters) and I needed the bits for
347	* tracking mappings of a page. 48-12 = 36. That leaves 28 bits, which means a
348	* decent range for the page id: 2^(28+12) = 1024TB.
349	*
350	* In additions to these, we'll have to keep maintaining the page flags as we
351	* currently do. Although it wouldn't harm to optimize these quite a bit, like
352	* for instance the ROM shouldn't depend on having a write handler installed
353	* in order for it to become read-only. A RO/RW bit should be considered so
354	* that the page syncing code doesn't have to mess about checking multiple
355	* flag combinations (ROM \|\| RW handler \|\| write monitored) in order to
356	* figure out how to setup a shadow PTE. But this of course, is second
357	* priority at present. Current this requires 12 bits, but could probably
358	* be optimized to ~8.
359	*
360	* Then there's the 24 bits used to track which shadow page tables are
361	* currently mapping a page for the purpose of speeding up physical
362	* access handlers, and thereby the page pool cache. More bit for this
363	* purpose wouldn't hurt IIRC.
364	*
365	* Then there is a new bit in which we need to record what kind of page
366	* this is, shared, zero, normal or write-monitored-normal. This'll
367	* require 2 bits. One bit might be needed for indicating whether a
368	* write monitored page has been written to. And yet another one or
369	* two for tracking migration status. 3-4 bits total then.
370	*
371	* Whatever is left will can be used to record the sharabilitiy of a
372	* page. The page checksum will not be stored in the per-VM table as
373	* the idle thread will not be permitted to do modifications to it.
374	* It will instead have to keep its own working set of potentially
375	* shareable pages and their check sums and stuff.
376	*
377	* For the present we'll keep the current packing of the
378	* PGMRAMRANGE::aHCPhys to keep the changes simple, only of course,
379	* we'll have to change it to a struct with a total of 128-bits at
380	* our disposal.
381	*
382	* The initial layout will be like this:
383	* @verbatim
384	RTHCPHYS HCPhys; The current stuff.
385	63:40 Current shadow PT tracking stuff.
386	39:12 The physical page frame number.
387	11:0 The current flags.
388	uint32_t u28PageId : 28; The page id.
389	uint32_t u2State : 2; The page state { zero, shared, normal, write monitored }.
390	uint32_t fWrittenTo : 1; Whether a write monitored page was written to.
391	uint32_t u1Reserved : 1; Reserved for later.
392	uint32_t u32Reserved; Reserved for later, mostly sharing stats.
393	@endverbatim
394	*
395	* The final layout will be something like this:
396	* @verbatim
397	RTHCPHYS HCPhys; The current stuff.
398	63:48 High page id (12+).
399	47:12 The physical page frame number.
400	11:0 Low page id.
401	uint32_t fReadOnly : 1; Whether it's readonly page (rom or monitored in some way).
402	uint32_t u3Type : 3; The page type {RESERVED, MMIO, MMIO2, ROM, shadowed ROM, RAM}.
403	uint32_t u2PhysMon : 2; Physical access handler type {none, read, write, all}.
404	uint32_t u2VirtMon : 2; Virtual access handler type {none, read, write, all}..
405	uint32_t u2State : 2; The page state { zero, shared, normal, write monitored }.
406	uint32_t fWrittenTo : 1; Whether a write monitored page was written to.
407	uint32_t u20Reserved : 20; Reserved for later, mostly sharing stats.
408	uint32_t u32Tracking; The shadow PT tracking stuff, roughly.
409	@endverbatim
410	*
411	* Cost wise, this means we'll double the cost for guest memory. There isn't anyway
412	* around that I'm afraid. It means that the cost of dealing out 32GB of memory
413	* to one or more VMs is: (32GB >> PAGE_SHIFT) * 16 bytes, or 128MBs. Or another
414	* example, the VM heap cost when assigning 1GB to a VM will be: 4MB.
415	*
416	* A couple of cost examples for the total cost per-VM + kernel.
417	* 32-bit Windows and 32-bit linux:
418	* 1GB guest ram, 256K pages: 4MB + 2MB(+) = 6MB
419	* 4GB guest ram, 1M pages: 16MB + 8MB(+) = 24MB
420	* 32GB guest ram, 8M pages: 128MB + 64MB(+) = 192MB
421	* 64-bit Windows and 64-bit linux:
422	* 1GB guest ram, 256K pages: 4MB + 3MB(+) = 7MB
423	* 4GB guest ram, 1M pages: 16MB + 12MB(+) = 28MB
424	* 32GB guest ram, 8M pages: 128MB + 96MB(+) = 224MB
425	*
426	* UPDATE - 2007-09-27:
427	* Will need a ballooned flag/state too because we cannot
428	* trust the guest 100% and reporting the same page as ballooned more
429	* than once will put the GMM off balance.
430	*
431	*
432	* @subsection subsec_pgmPhys_Serializing Serializing Access
433	*
434	* Initially, we'll try a simple scheme:
435	*
436	* - The per-VM RAM tracking structures (PGMRAMRANGE) is only modified
437	* by the EMT thread of that VM while in the pgm critsect.
438	* - Other threads in the VM process that needs to make reliable use of
439	* the per-VM RAM tracking structures will enter the critsect.
440	* - No process external thread or kernel thread will ever try enter
441	* the pgm critical section, as that just won't work.
442	* - The idle thread (and similar threads) doesn't not need 100% reliable
443	* data when performing it tasks as the EMT thread will be the one to
444	* do the actual changes later anyway. So, as long as it only accesses
445	* the main ram range, it can do so by somehow preventing the VM from
446	* being destroyed while it works on it...
447	*
448	* - The over-commitment management, including the allocating/freeing
449	* chunks, is serialized by a ring-0 mutex lock (a fast one since the
450	* more mundane mutex implementation is broken on Linux).
451	* - A separate mutex is protecting the set of allocation chunks so
452	* that pages can be shared or/and freed up while some other VM is
453	* allocating more chunks. This mutex can be take from under the other
454	* one, but not the other way around.
455	*
456	*
457	* @subsection subsec_pgmPhys_Request VM Request interface
458	*
459	* When in ring-0 it will become necessary to send requests to a VM so it can
460	* for instance move a page while defragmenting during VM destroy. The idle
461	* thread will make use of this interface to request VMs to setup shared
462	* pages and to perform write monitoring of pages.
463	*
464	* I would propose an interface similar to the current VMReq interface, similar
465	* in that it doesn't require locking and that the one sending the request may
466	* wait for completion if it wishes to. This shouldn't be very difficult to
467	* realize.
468	*
469	* The requests themselves are also pretty simple. They are basically:
470	* -# Check that some precondition is still true.
471	* -# Do the update.
472	* -# Update all shadow page tables involved with the page.
473	*
474	* The 3rd step is identical to what we're already doing when updating a
475	* physical handler, see pgmHandlerPhysicalSetRamFlagsAndFlushShadowPTs.
476	*
477	*
478	*
479	* @section sec_pgmPhys_MappingCaches Mapping Caches
480	*
481	* In order to be able to map in and out memory and to be able to support
482	* guest with more RAM than we've got virtual address space, we'll employing
483	* a mapping cache. Normally ring-0 and ring-3 can share the same cache,
484	* however on 32-bit darwin the ring-0 code is running in a different memory
485	* context and therefore needs a separate cache. In raw-mode context we also
486	* need a separate cache. The 32-bit darwin mapping cache and the one for
487	* raw-mode context share a lot of code, see PGMRZDYNMAP.
488	*
489	*
490	* @subsection subsec_pgmPhys_MappingCaches_R3 Ring-3
491	*
492	* We've considered implementing the ring-3 mapping cache page based but found
493	* that this was bother some when one had to take into account TLBs+SMP and
494	* portability (missing the necessary APIs on several platforms). There were
495	* also some performance concerns with this approach which hadn't quite been
496	* worked out.
497	*
498	* Instead, we'll be mapping allocation chunks into the VM process. This simplifies
499	* matters greatly quite a bit since we don't need to invent any new ring-0 stuff,
500	* only some minor RTR0MEMOBJ mapping stuff. The main concern here is that mapping
501	* compared to the previous idea is that mapping or unmapping a 1MB chunk is more
502	* costly than a single page, although how much more costly is uncertain. We'll
503	* try address this by using a very big cache, preferably bigger than the actual
504	* VM RAM size if possible. The current VM RAM sizes should give some idea for
505	* 32-bit boxes, while on 64-bit we can probably get away with employing an
506	* unlimited cache.
507	*
508	* The cache have to parts, as already indicated, the ring-3 side and the
509	* ring-0 side.
510	*
511	* The ring-0 will be tied to the page allocator since it will operate on the
512	* memory objects it contains. It will therefore require the first ring-0 mutex
513	* discussed in @ref subsec_pgmPhys_Serializing. We
514	* some double house keeping wrt to who has mapped what I think, since both
515	* VMMR0.r0 and RTR0MemObj will keep track of mapping relations
516	*
517	* The ring-3 part will be protected by the pgm critsect. For simplicity, we'll
518	* require anyone that desires to do changes to the mapping cache to do that
519	* from within this critsect. Alternatively, we could employ a separate critsect
520	* for serializing changes to the mapping cache as this would reduce potential
521	* contention with other threads accessing mappings unrelated to the changes
522	* that are in process. We can see about this later, contention will show
523	* up in the statistics anyway, so it'll be simple to tell.
524	*
525	* The organization of the ring-3 part will be very much like how the allocation
526	* chunks are organized in ring-0, that is in an AVL tree by chunk id. To avoid
527	* having to walk the tree all the time, we'll have a couple of lookaside entries
528	* like in we do for I/O ports and MMIO in IOM.
529	*
530	* The simplified flow of a PGMPhysRead/Write function:
531	* -# Enter the PGM critsect.
532	* -# Lookup GCPhys in the ram ranges and get the Page ID.
533	* -# Calc the Allocation Chunk ID from the Page ID.
534	* -# Check the lookaside entries and then the AVL tree for the Chunk ID.
535	* If not found in cache:
536	* -# Call ring-0 and request it to be mapped and supply
537	* a chunk to be unmapped if the cache is maxed out already.
538	* -# Insert the new mapping into the AVL tree (id + R3 address).
539	* -# Update the relevant lookaside entry and return the mapping address.
540	* -# Do the read/write according to monitoring flags and everything.
541	* -# Leave the critsect.
542	*
543	*
544	* @section sec_pgmPhys_Fallback Fallback
545	*
546	* Current all the "second tier" hosts will not support the RTR0MemObjAllocPhysNC
547	* API and thus require a fallback.
548	*
549	* So, when RTR0MemObjAllocPhysNC returns VERR_NOT_SUPPORTED the page allocator
550	* will return to the ring-3 caller (and later ring-0) and asking it to seed
551	* the page allocator with some fresh pages (VERR_GMM_SEED_ME). Ring-3 will
552	* then perform an SUPR3PageAlloc(cbChunk >> PAGE_SHIFT) call and make a
553	* "SeededAllocPages" call to ring-0.
554	*
555	* The first time ring-0 sees the VERR_NOT_SUPPORTED failure it will disable
556	* all page sharing (zero page detection will continue). It will also force
557	* all allocations to come from the VM which seeded the page. Both these
558	* measures are taken to make sure that there will never be any need for
559	* mapping anything into ring-3 - everything will be mapped already.
560	*
561	* Whether we'll continue to use the current MM locked memory management
562	* for this I don't quite know (I'd prefer not to and just ditch that all
563	* together), we'll see what's simplest to do.
564	*
565	*
566	*
567	* @section sec_pgmPhys_Changes Changes
568	*
569	* Breakdown of the changes involved?
570	*/
571
572	/*******************************************************************************
573	* Header Files *
574	*******************************************************************************/
575	#define LOG_GROUP LOG_GROUP_PGM
576	#include <VBox/vmm/dbgf.h>
577	#include <VBox/vmm/pgm.h>
578	#include <VBox/vmm/cpum.h>
579	#include <VBox/vmm/iom.h>
580	#include <VBox/sup.h>
581	#include <VBox/vmm/mm.h>
582	#include <VBox/vmm/em.h>
583	#include <VBox/vmm/stam.h>
584	#include <VBox/vmm/rem.h>
585	#include <VBox/vmm/selm.h>
586	#include <VBox/vmm/ssm.h>
587	#include <VBox/vmm/hwaccm.h>
588	#include "PGMInternal.h"
589	#include <VBox/vmm/vm.h>
590	#include "PGMInline.h"
591
592	#include <VBox/dbg.h>
593	#include <VBox/param.h>
594	#include <VBox/err.h>
595
596	#include <iprt/asm.h>
597	#include <iprt/asm-amd64-x86.h>
598	#include <iprt/assert.h>
599	#include <iprt/env.h>
600	#include <iprt/mem.h>
601	#include <iprt/file.h>
602	#include <iprt/string.h>
603	#include <iprt/thread.h>
604
605
606	/*******************************************************************************
607	* Internal Functions *
608	*******************************************************************************/
609	static int pgmR3InitPaging(PVM pVM);
610	static int pgmR3InitStats(PVM pVM);
611	static DECLCALLBACK(void) pgmR3PhysInfo(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs);
612	static DECLCALLBACK(void) pgmR3InfoMode(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs);
613	static DECLCALLBACK(void) pgmR3InfoCr3(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs);
614	static DECLCALLBACK(int) pgmR3RelocatePhysHandler(PAVLROGCPHYSNODECORE pNode, void *pvUser);
615	static DECLCALLBACK(int) pgmR3RelocateVirtHandler(PAVLROGCPTRNODECORE pNode, void *pvUser);
616	static DECLCALLBACK(int) pgmR3RelocateHyperVirtHandler(PAVLROGCPTRNODECORE pNode, void *pvUser);
617	#ifdef VBOX_STRICT
618	static DECLCALLBACK(void) pgmR3ResetNoMorePhysWritesFlag(PVM pVM, VMSTATE enmState, VMSTATE enmOldState, void *pvUser);
619	#endif
620	static int pgmR3ModeDataInit(PVM pVM, bool fResolveGCAndR0);
621	static void pgmR3ModeDataSwitch(PVM pVM, PVMCPU pVCpu, PGMMODE enmShw, PGMMODE enmGst);
622	static PGMMODE pgmR3CalcShadowMode(PVM pVM, PGMMODE enmGuestMode, SUPPAGINGMODE enmHostMode, PGMMODE enmShadowMode, VMMSWITCHER *penmSwitcher);
623
624	#ifdef VBOX_WITH_DEBUGGER
625	/** @todo Convert the first two commands to 'info' items. */
626	static DECLCALLBACK(int) pgmR3CmdRam(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs);
627	static DECLCALLBACK(int) pgmR3CmdError(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs);
628	static DECLCALLBACK(int) pgmR3CmdSync(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs);
629	static DECLCALLBACK(int) pgmR3CmdSyncAlways(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs);
630	# ifdef VBOX_STRICT
631	static DECLCALLBACK(int) pgmR3CmdAssertCR3(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs);
632	# endif
633	static DECLCALLBACK(int) pgmR3CmdPhysToFile(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs);
634	#endif
635
636
637	/*******************************************************************************
638	* Global Variables *
639	*******************************************************************************/
640	#ifdef VBOX_WITH_DEBUGGER
641	/** Argument descriptors for '.pgmerror' and '.pgmerroroff'. */
642	static const DBGCVARDESC g_aPgmErrorArgs[] =
643	{
644	/* cTimesMin, cTimesMax, enmCategory, fFlags, pszName, pszDescription */
645	{ 0, 1, DBGCVAR_CAT_STRING, 0, "where", "Error injection location." },
646	};
647
648	static const DBGCVARDESC g_aPgmPhysToFileArgs[] =
649	{
650	/* cTimesMin, cTimesMax, enmCategory, fFlags, pszName, pszDescription */
651	{ 1, 1, DBGCVAR_CAT_STRING, 0, "file", "The file name." },
652	{ 0, 1, DBGCVAR_CAT_STRING, 0, "nozero", "If present, zero pages are skipped." },
653	};
654
655	# ifdef DEBUG_sandervl
656	static const DBGCVARDESC g_aPgmCountPhysWritesArgs[] =
657	{
658	/* cTimesMin, cTimesMax, enmCategory, fFlags, pszName, pszDescription */
659	{ 1, 1, DBGCVAR_CAT_STRING, 0, "enabled", "on/off." },
660	{ 1, 1, DBGCVAR_CAT_NUMBER_NO_RANGE, 0, "interval", "Interval in ms." },
661	};
662	# endif
663
664	/** Command descriptors. */
665	static const DBGCCMD g_aCmds[] =
666	{
667	/* pszCmd, cArgsMin, cArgsMax, paArgDesc, cArgDescs, fFlags, pfnHandler pszSyntax, ....pszDescription */
668	{ "pgmram", 0, 0, NULL, 0, 0, pgmR3CmdRam, "", "Display the ram ranges." },
669	{ "pgmsync", 0, 0, NULL, 0, 0, pgmR3CmdSync, "", "Sync the CR3 page." },
670	{ "pgmerror", 0, 1, &g_aPgmErrorArgs[0], 1, 0, pgmR3CmdError, "", "Enables inject runtime of errors into parts of PGM." },
671	{ "pgmerroroff", 0, 1, &g_aPgmErrorArgs[0], 1, 0, pgmR3CmdError, "", "Disables inject runtime errors into parts of PGM." },
672	# ifdef VBOX_STRICT
673	{ "pgmassertcr3", 0, 0, NULL, 0, 0, pgmR3CmdAssertCR3, "", "Check the shadow CR3 mapping." },
674	# if HC_ARCH_BITS == 64
675	{ "pgmcheckduppages", 0, 0, NULL, 0, 0, pgmR3CmdCheckDuplicatePages, "", "Check for duplicate pages in all running VMs." },
676	{ "pgmsharedmodules", 0, 0, NULL, 0, 0, pgmR3CmdShowSharedModules, "", "Print shared modules info." },
677	# endif
678	# endif
679	{ "pgmsyncalways", 0, 0, NULL, 0, 0, pgmR3CmdSyncAlways, "", "Toggle permanent CR3 syncing." },
680	{ "pgmphystofile", 1, 2, &g_aPgmPhysToFileArgs[0], 2, 0, pgmR3CmdPhysToFile, "", "Save the physical memory to file." },
681	};
682	#endif
683
684
685
686
687	/*
688	* Shadow - 32-bit mode
689	*/
690	#define PGM_SHW_TYPE PGM_TYPE_32BIT
691	#define PGM_SHW_NAME(name) PGM_SHW_NAME_32BIT(name)
692	#define PGM_SHW_NAME_RC_STR(name) PGM_SHW_NAME_RC_32BIT_STR(name)
693	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_32BIT_STR(name)
694	#include "PGMShw.h"
695
696	/* Guest - real mode */
697	#define PGM_GST_TYPE PGM_TYPE_REAL
698	#define PGM_GST_NAME(name) PGM_GST_NAME_REAL(name)
699	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_REAL_STR(name)
700	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_REAL_STR(name)
701	#define PGM_BTH_NAME(name) PGM_BTH_NAME_32BIT_REAL(name)
702	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_32BIT_REAL_STR(name)
703	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_32BIT_REAL_STR(name)
704	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_32BIT_PT_FOR_PHYS
705	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_32BIT_PD_PHYS
706	#include "PGMBth.h"
707	#include "PGMGstDefs.h"
708	#include "PGMGst.h"
709	#undef BTH_PGMPOOLKIND_PT_FOR_PT
710	#undef BTH_PGMPOOLKIND_ROOT
711	#undef PGM_BTH_NAME
712	#undef PGM_BTH_NAME_RC_STR
713	#undef PGM_BTH_NAME_R0_STR
714	#undef PGM_GST_TYPE
715	#undef PGM_GST_NAME
716	#undef PGM_GST_NAME_RC_STR
717	#undef PGM_GST_NAME_R0_STR
718
719	/* Guest - protected mode */
720	#define PGM_GST_TYPE PGM_TYPE_PROT
721	#define PGM_GST_NAME(name) PGM_GST_NAME_PROT(name)
722	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PROT_STR(name)
723	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PROT_STR(name)
724	#define PGM_BTH_NAME(name) PGM_BTH_NAME_32BIT_PROT(name)
725	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_32BIT_PROT_STR(name)
726	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_32BIT_PROT_STR(name)
727	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_32BIT_PT_FOR_PHYS
728	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_32BIT_PD_PHYS
729	#include "PGMBth.h"
730	#include "PGMGstDefs.h"
731	#include "PGMGst.h"
732	#undef BTH_PGMPOOLKIND_PT_FOR_PT
733	#undef BTH_PGMPOOLKIND_ROOT
734	#undef PGM_BTH_NAME
735	#undef PGM_BTH_NAME_RC_STR
736	#undef PGM_BTH_NAME_R0_STR
737	#undef PGM_GST_TYPE
738	#undef PGM_GST_NAME
739	#undef PGM_GST_NAME_RC_STR
740	#undef PGM_GST_NAME_R0_STR
741
742	/* Guest - 32-bit mode */
743	#define PGM_GST_TYPE PGM_TYPE_32BIT
744	#define PGM_GST_NAME(name) PGM_GST_NAME_32BIT(name)
745	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_32BIT_STR(name)
746	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_32BIT_STR(name)
747	#define PGM_BTH_NAME(name) PGM_BTH_NAME_32BIT_32BIT(name)
748	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_32BIT_32BIT_STR(name)
749	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_32BIT_32BIT_STR(name)
750	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_32BIT_PT_FOR_32BIT_PT
751	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_32BIT_PT_FOR_32BIT_4MB
752	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_32BIT_PD
753	#include "PGMBth.h"
754	#include "PGMGstDefs.h"
755	#include "PGMGst.h"
756	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
757	#undef BTH_PGMPOOLKIND_PT_FOR_PT
758	#undef BTH_PGMPOOLKIND_ROOT
759	#undef PGM_BTH_NAME
760	#undef PGM_BTH_NAME_RC_STR
761	#undef PGM_BTH_NAME_R0_STR
762	#undef PGM_GST_TYPE
763	#undef PGM_GST_NAME
764	#undef PGM_GST_NAME_RC_STR
765	#undef PGM_GST_NAME_R0_STR
766
767	#undef PGM_SHW_TYPE
768	#undef PGM_SHW_NAME
769	#undef PGM_SHW_NAME_RC_STR
770	#undef PGM_SHW_NAME_R0_STR
771
772
773	/*
774	* Shadow - PAE mode
775	*/
776	#define PGM_SHW_TYPE PGM_TYPE_PAE
777	#define PGM_SHW_NAME(name) PGM_SHW_NAME_PAE(name)
778	#define PGM_SHW_NAME_RC_STR(name) PGM_SHW_NAME_RC_PAE_STR(name)
779	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_PAE_STR(name)
780	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_REAL(name)
781	#include "PGMShw.h"
782
783	/* Guest - real mode */
784	#define PGM_GST_TYPE PGM_TYPE_REAL
785	#define PGM_GST_NAME(name) PGM_GST_NAME_REAL(name)
786	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_REAL_STR(name)
787	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_REAL_STR(name)
788	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_REAL(name)
789	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_PAE_REAL_STR(name)
790	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_PAE_REAL_STR(name)
791	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
792	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_PAE_PDPT_PHYS
793	#include "PGMGstDefs.h"
794	#include "PGMBth.h"
795	#undef BTH_PGMPOOLKIND_PT_FOR_PT
796	#undef BTH_PGMPOOLKIND_ROOT
797	#undef PGM_BTH_NAME
798	#undef PGM_BTH_NAME_RC_STR
799	#undef PGM_BTH_NAME_R0_STR
800	#undef PGM_GST_TYPE
801	#undef PGM_GST_NAME
802	#undef PGM_GST_NAME_RC_STR
803	#undef PGM_GST_NAME_R0_STR
804
805	/* Guest - protected mode */
806	#define PGM_GST_TYPE PGM_TYPE_PROT
807	#define PGM_GST_NAME(name) PGM_GST_NAME_PROT(name)
808	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PROT_STR(name)
809	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PROT_STR(name)
810	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_PROT(name)
811	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_PAE_PROT_STR(name)
812	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_PAE_PROT_STR(name)
813	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
814	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_PAE_PDPT_PHYS
815	#include "PGMGstDefs.h"
816	#include "PGMBth.h"
817	#undef BTH_PGMPOOLKIND_PT_FOR_PT
818	#undef BTH_PGMPOOLKIND_ROOT
819	#undef PGM_BTH_NAME
820	#undef PGM_BTH_NAME_RC_STR
821	#undef PGM_BTH_NAME_R0_STR
822	#undef PGM_GST_TYPE
823	#undef PGM_GST_NAME
824	#undef PGM_GST_NAME_RC_STR
825	#undef PGM_GST_NAME_R0_STR
826
827	/* Guest - 32-bit mode */
828	#define PGM_GST_TYPE PGM_TYPE_32BIT
829	#define PGM_GST_NAME(name) PGM_GST_NAME_32BIT(name)
830	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_32BIT_STR(name)
831	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_32BIT_STR(name)
832	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_32BIT(name)
833	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_PAE_32BIT_STR(name)
834	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_PAE_32BIT_STR(name)
835	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_32BIT_PT
836	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_32BIT_4MB
837	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_PAE_PDPT_FOR_32BIT
838	#include "PGMGstDefs.h"
839	#include "PGMBth.h"
840	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
841	#undef BTH_PGMPOOLKIND_PT_FOR_PT
842	#undef BTH_PGMPOOLKIND_ROOT
843	#undef PGM_BTH_NAME
844	#undef PGM_BTH_NAME_RC_STR
845	#undef PGM_BTH_NAME_R0_STR
846	#undef PGM_GST_TYPE
847	#undef PGM_GST_NAME
848	#undef PGM_GST_NAME_RC_STR
849	#undef PGM_GST_NAME_R0_STR
850
851	/* Guest - PAE mode */
852	#define PGM_GST_TYPE PGM_TYPE_PAE
853	#define PGM_GST_NAME(name) PGM_GST_NAME_PAE(name)
854	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PAE_STR(name)
855	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PAE_STR(name)
856	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_PAE(name)
857	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_PAE_PAE_STR(name)
858	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_PAE_PAE_STR(name)
859	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
860	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
861	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_PAE_PDPT
862	#include "PGMBth.h"
863	#include "PGMGstDefs.h"
864	#include "PGMGst.h"
865	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
866	#undef BTH_PGMPOOLKIND_PT_FOR_PT
867	#undef BTH_PGMPOOLKIND_ROOT
868	#undef PGM_BTH_NAME
869	#undef PGM_BTH_NAME_RC_STR
870	#undef PGM_BTH_NAME_R0_STR
871	#undef PGM_GST_TYPE
872	#undef PGM_GST_NAME
873	#undef PGM_GST_NAME_RC_STR
874	#undef PGM_GST_NAME_R0_STR
875
876	#undef PGM_SHW_TYPE
877	#undef PGM_SHW_NAME
878	#undef PGM_SHW_NAME_RC_STR
879	#undef PGM_SHW_NAME_R0_STR
880
881
882	/*
883	* Shadow - AMD64 mode
884	*/
885	#define PGM_SHW_TYPE PGM_TYPE_AMD64
886	#define PGM_SHW_NAME(name) PGM_SHW_NAME_AMD64(name)
887	#define PGM_SHW_NAME_RC_STR(name) PGM_SHW_NAME_RC_AMD64_STR(name)
888	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_AMD64_STR(name)
889	#include "PGMShw.h"
890
891	#ifdef VBOX_WITH_64_BITS_GUESTS
892	/* Guest - AMD64 mode */
893	# define PGM_GST_TYPE PGM_TYPE_AMD64
894	# define PGM_GST_NAME(name) PGM_GST_NAME_AMD64(name)
895	# define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_AMD64_STR(name)
896	# define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_AMD64_STR(name)
897	# define PGM_BTH_NAME(name) PGM_BTH_NAME_AMD64_AMD64(name)
898	# define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_AMD64_AMD64_STR(name)
899	# define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_AMD64_AMD64_STR(name)
900	# define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
901	# define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
902	# define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_64BIT_PML4
903	# include "PGMBth.h"
904	# include "PGMGstDefs.h"
905	# include "PGMGst.h"
906	# undef BTH_PGMPOOLKIND_PT_FOR_BIG
907	# undef BTH_PGMPOOLKIND_PT_FOR_PT
908	# undef BTH_PGMPOOLKIND_ROOT
909	# undef PGM_BTH_NAME
910	# undef PGM_BTH_NAME_RC_STR
911	# undef PGM_BTH_NAME_R0_STR
912	# undef PGM_GST_TYPE
913	# undef PGM_GST_NAME
914	# undef PGM_GST_NAME_RC_STR
915	# undef PGM_GST_NAME_R0_STR
916	#endif /* VBOX_WITH_64_BITS_GUESTS */
917
918	#undef PGM_SHW_TYPE
919	#undef PGM_SHW_NAME
920	#undef PGM_SHW_NAME_RC_STR
921	#undef PGM_SHW_NAME_R0_STR
922
923
924	/*
925	* Shadow - Nested paging mode
926	*/
927	#define PGM_SHW_TYPE PGM_TYPE_NESTED
928	#define PGM_SHW_NAME(name) PGM_SHW_NAME_NESTED(name)
929	#define PGM_SHW_NAME_RC_STR(name) PGM_SHW_NAME_RC_NESTED_STR(name)
930	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_NESTED_STR(name)
931	#include "PGMShw.h"
932
933	/* Guest - real mode */
934	#define PGM_GST_TYPE PGM_TYPE_REAL
935	#define PGM_GST_NAME(name) PGM_GST_NAME_REAL(name)
936	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_REAL_STR(name)
937	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_REAL_STR(name)
938	#define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_REAL(name)
939	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_NESTED_REAL_STR(name)
940	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_REAL_STR(name)
941	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
942	#include "PGMGstDefs.h"
943	#include "PGMBth.h"
944	#undef BTH_PGMPOOLKIND_PT_FOR_PT
945	#undef PGM_BTH_NAME
946	#undef PGM_BTH_NAME_RC_STR
947	#undef PGM_BTH_NAME_R0_STR
948	#undef PGM_GST_TYPE
949	#undef PGM_GST_NAME
950	#undef PGM_GST_NAME_RC_STR
951	#undef PGM_GST_NAME_R0_STR
952
953	/* Guest - protected mode */
954	#define PGM_GST_TYPE PGM_TYPE_PROT
955	#define PGM_GST_NAME(name) PGM_GST_NAME_PROT(name)
956	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PROT_STR(name)
957	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PROT_STR(name)
958	#define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_PROT(name)
959	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_NESTED_PROT_STR(name)
960	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_PROT_STR(name)
961	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
962	#include "PGMGstDefs.h"
963	#include "PGMBth.h"
964	#undef BTH_PGMPOOLKIND_PT_FOR_PT
965	#undef PGM_BTH_NAME
966	#undef PGM_BTH_NAME_RC_STR
967	#undef PGM_BTH_NAME_R0_STR
968	#undef PGM_GST_TYPE
969	#undef PGM_GST_NAME
970	#undef PGM_GST_NAME_RC_STR
971	#undef PGM_GST_NAME_R0_STR
972
973	/* Guest - 32-bit mode */
974	#define PGM_GST_TYPE PGM_TYPE_32BIT
975	#define PGM_GST_NAME(name) PGM_GST_NAME_32BIT(name)
976	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_32BIT_STR(name)
977	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_32BIT_STR(name)
978	#define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_32BIT(name)
979	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_NESTED_32BIT_STR(name)
980	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_32BIT_STR(name)
981	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_32BIT_PT
982	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_32BIT_4MB
983	#include "PGMGstDefs.h"
984	#include "PGMBth.h"
985	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
986	#undef BTH_PGMPOOLKIND_PT_FOR_PT
987	#undef PGM_BTH_NAME
988	#undef PGM_BTH_NAME_RC_STR
989	#undef PGM_BTH_NAME_R0_STR
990	#undef PGM_GST_TYPE
991	#undef PGM_GST_NAME
992	#undef PGM_GST_NAME_RC_STR
993	#undef PGM_GST_NAME_R0_STR
994
995	/* Guest - PAE mode */
996	#define PGM_GST_TYPE PGM_TYPE_PAE
997	#define PGM_GST_NAME(name) PGM_GST_NAME_PAE(name)
998	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PAE_STR(name)
999	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PAE_STR(name)
1000	#define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_PAE(name)
1001	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_NESTED_PAE_STR(name)
1002	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_PAE_STR(name)
1003	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
1004	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
1005	#include "PGMGstDefs.h"
1006	#include "PGMBth.h"
1007	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
1008	#undef BTH_PGMPOOLKIND_PT_FOR_PT
1009	#undef PGM_BTH_NAME
1010	#undef PGM_BTH_NAME_RC_STR
1011	#undef PGM_BTH_NAME_R0_STR
1012	#undef PGM_GST_TYPE
1013	#undef PGM_GST_NAME
1014	#undef PGM_GST_NAME_RC_STR
1015	#undef PGM_GST_NAME_R0_STR
1016
1017	#ifdef VBOX_WITH_64_BITS_GUESTS
1018	/* Guest - AMD64 mode */
1019	# define PGM_GST_TYPE PGM_TYPE_AMD64
1020	# define PGM_GST_NAME(name) PGM_GST_NAME_AMD64(name)
1021	# define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_AMD64_STR(name)
1022	# define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_AMD64_STR(name)
1023	# define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_AMD64(name)
1024	# define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_NESTED_AMD64_STR(name)
1025	# define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_AMD64_STR(name)
1026	# define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
1027	# define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
1028	# include "PGMGstDefs.h"
1029	# include "PGMBth.h"
1030	# undef BTH_PGMPOOLKIND_PT_FOR_BIG
1031	# undef BTH_PGMPOOLKIND_PT_FOR_PT
1032	# undef PGM_BTH_NAME
1033	# undef PGM_BTH_NAME_RC_STR
1034	# undef PGM_BTH_NAME_R0_STR
1035	# undef PGM_GST_TYPE
1036	# undef PGM_GST_NAME
1037	# undef PGM_GST_NAME_RC_STR
1038	# undef PGM_GST_NAME_R0_STR
1039	#endif /* VBOX_WITH_64_BITS_GUESTS */
1040
1041	#undef PGM_SHW_TYPE
1042	#undef PGM_SHW_NAME
1043	#undef PGM_SHW_NAME_RC_STR
1044	#undef PGM_SHW_NAME_R0_STR
1045
1046
1047	/*
1048	* Shadow - EPT
1049	*/
1050	#define PGM_SHW_TYPE PGM_TYPE_EPT
1051	#define PGM_SHW_NAME(name) PGM_SHW_NAME_EPT(name)
1052	#define PGM_SHW_NAME_RC_STR(name) PGM_SHW_NAME_RC_EPT_STR(name)
1053	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_EPT_STR(name)
1054	#include "PGMShw.h"
1055
1056	/* Guest - real mode */
1057	#define PGM_GST_TYPE PGM_TYPE_REAL
1058	#define PGM_GST_NAME(name) PGM_GST_NAME_REAL(name)
1059	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_REAL_STR(name)
1060	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_REAL_STR(name)
1061	#define PGM_BTH_NAME(name) PGM_BTH_NAME_EPT_REAL(name)
1062	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_EPT_REAL_STR(name)
1063	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_EPT_REAL_STR(name)
1064	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
1065	#include "PGMGstDefs.h"
1066	#include "PGMBth.h"
1067	#undef BTH_PGMPOOLKIND_PT_FOR_PT
1068	#undef PGM_BTH_NAME
1069	#undef PGM_BTH_NAME_RC_STR
1070	#undef PGM_BTH_NAME_R0_STR
1071	#undef PGM_GST_TYPE
1072	#undef PGM_GST_NAME
1073	#undef PGM_GST_NAME_RC_STR
1074	#undef PGM_GST_NAME_R0_STR
1075
1076	/* Guest - protected mode */
1077	#define PGM_GST_TYPE PGM_TYPE_PROT
1078	#define PGM_GST_NAME(name) PGM_GST_NAME_PROT(name)
1079	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PROT_STR(name)
1080	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PROT_STR(name)
1081	#define PGM_BTH_NAME(name) PGM_BTH_NAME_EPT_PROT(name)
1082	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_EPT_PROT_STR(name)
1083	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_EPT_PROT_STR(name)
1084	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
1085	#include "PGMGstDefs.h"
1086	#include "PGMBth.h"
1087	#undef BTH_PGMPOOLKIND_PT_FOR_PT
1088	#undef PGM_BTH_NAME
1089	#undef PGM_BTH_NAME_RC_STR
1090	#undef PGM_BTH_NAME_R0_STR
1091	#undef PGM_GST_TYPE
1092	#undef PGM_GST_NAME
1093	#undef PGM_GST_NAME_RC_STR
1094	#undef PGM_GST_NAME_R0_STR
1095
1096	/* Guest - 32-bit mode */
1097	#define PGM_GST_TYPE PGM_TYPE_32BIT
1098	#define PGM_GST_NAME(name) PGM_GST_NAME_32BIT(name)
1099	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_32BIT_STR(name)
1100	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_32BIT_STR(name)
1101	#define PGM_BTH_NAME(name) PGM_BTH_NAME_EPT_32BIT(name)
1102	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_EPT_32BIT_STR(name)
1103	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_EPT_32BIT_STR(name)
1104	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_32BIT_PT
1105	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_32BIT_4MB
1106	#include "PGMGstDefs.h"
1107	#include "PGMBth.h"
1108	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
1109	#undef BTH_PGMPOOLKIND_PT_FOR_PT
1110	#undef PGM_BTH_NAME
1111	#undef PGM_BTH_NAME_RC_STR
1112	#undef PGM_BTH_NAME_R0_STR
1113	#undef PGM_GST_TYPE
1114	#undef PGM_GST_NAME
1115	#undef PGM_GST_NAME_RC_STR
1116	#undef PGM_GST_NAME_R0_STR
1117
1118	/* Guest - PAE mode */
1119	#define PGM_GST_TYPE PGM_TYPE_PAE
1120	#define PGM_GST_NAME(name) PGM_GST_NAME_PAE(name)
1121	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PAE_STR(name)
1122	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PAE_STR(name)
1123	#define PGM_BTH_NAME(name) PGM_BTH_NAME_EPT_PAE(name)
1124	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_EPT_PAE_STR(name)
1125	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_EPT_PAE_STR(name)
1126	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
1127	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
1128	#include "PGMGstDefs.h"
1129	#include "PGMBth.h"
1130	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
1131	#undef BTH_PGMPOOLKIND_PT_FOR_PT
1132	#undef PGM_BTH_NAME
1133	#undef PGM_BTH_NAME_RC_STR
1134	#undef PGM_BTH_NAME_R0_STR
1135	#undef PGM_GST_TYPE
1136	#undef PGM_GST_NAME
1137	#undef PGM_GST_NAME_RC_STR
1138	#undef PGM_GST_NAME_R0_STR
1139
1140	#ifdef VBOX_WITH_64_BITS_GUESTS
1141	/* Guest - AMD64 mode */
1142	# define PGM_GST_TYPE PGM_TYPE_AMD64
1143	# define PGM_GST_NAME(name) PGM_GST_NAME_AMD64(name)
1144	# define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_AMD64_STR(name)
1145	# define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_AMD64_STR(name)
1146	# define PGM_BTH_NAME(name) PGM_BTH_NAME_EPT_AMD64(name)
1147	# define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_EPT_AMD64_STR(name)
1148	# define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_EPT_AMD64_STR(name)
1149	# define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
1150	# define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
1151	# include "PGMGstDefs.h"
1152	# include "PGMBth.h"
1153	# undef BTH_PGMPOOLKIND_PT_FOR_BIG
1154	# undef BTH_PGMPOOLKIND_PT_FOR_PT
1155	# undef PGM_BTH_NAME
1156	# undef PGM_BTH_NAME_RC_STR
1157	# undef PGM_BTH_NAME_R0_STR
1158	# undef PGM_GST_TYPE
1159	# undef PGM_GST_NAME
1160	# undef PGM_GST_NAME_RC_STR
1161	# undef PGM_GST_NAME_R0_STR
1162	#endif /* VBOX_WITH_64_BITS_GUESTS */
1163
1164	#undef PGM_SHW_TYPE
1165	#undef PGM_SHW_NAME
1166	#undef PGM_SHW_NAME_RC_STR
1167	#undef PGM_SHW_NAME_R0_STR
1168
1169
1170
1171	/**
1172	* Initiates the paging of VM.
1173	*
1174	* @returns VBox status code.
1175	* @param pVM Pointer to VM structure.
1176	*/
1177	VMMR3DECL(int) PGMR3Init(PVM pVM)
1178	{
1179	LogFlow(("PGMR3Init:\n"));
1180	PCFGMNODE pCfgPGM = CFGMR3GetChild(CFGMR3GetRoot(pVM), "/PGM");
1181	int rc;
1182
1183	/*
1184	* Assert alignment and sizes.
1185	*/
1186	AssertCompile(sizeof(pVM->pgm.s) <= sizeof(pVM->pgm.padding));
1187	AssertCompile(sizeof(pVM->aCpus[0].pgm.s) <= sizeof(pVM->aCpus[0].pgm.padding));
1188	AssertCompileMemberAlignment(PGM, CritSect, sizeof(uintptr_t));
1189
1190	/*
1191	* Init the structure.
1192	*/
1193	#ifdef PGM_WITHOUT_MAPPINGS
1194	pVM->pgm.s.fMappingsDisabled = true;
1195	#endif
1196	pVM->pgm.s.offVM = RT_OFFSETOF(VM, pgm.s);
1197	pVM->pgm.s.offVCpuPGM = RT_OFFSETOF(VMCPU, pgm.s);
1198
1199	/* Init the per-CPU part. */
1200	for (VMCPUID idCpu = 0; idCpu < pVM->cCpus; idCpu++)
1201	{
1202	PVMCPU pVCpu = &pVM->aCpus[idCpu];
1203	PPGMCPU pPGM = &pVCpu->pgm.s;
1204
1205	pPGM->offVM = (uintptr_t)&pVCpu->pgm.s - (uintptr_t)pVM;
1206	pPGM->offVCpu = RT_OFFSETOF(VMCPU, pgm.s);
1207	pPGM->offPGM = (uintptr_t)&pVCpu->pgm.s - (uintptr_t)&pVM->pgm.s;
1208
1209	pPGM->enmShadowMode = PGMMODE_INVALID;
1210	pPGM->enmGuestMode = PGMMODE_INVALID;
1211
1212	pPGM->GCPhysCR3 = NIL_RTGCPHYS;
1213
1214	pPGM->pGst32BitPdR3 = NULL;
1215	pPGM->pGstPaePdptR3 = NULL;
1216	pPGM->pGstAmd64Pml4R3 = NULL;
1217	#ifndef VBOX_WITH_2X_4GB_ADDR_SPACE
1218	pPGM->pGst32BitPdR0 = NIL_RTR0PTR;
1219	pPGM->pGstPaePdptR0 = NIL_RTR0PTR;
1220	pPGM->pGstAmd64Pml4R0 = NIL_RTR0PTR;
1221	#endif
1222	pPGM->pGst32BitPdRC = NIL_RTRCPTR;
1223	pPGM->pGstPaePdptRC = NIL_RTRCPTR;
1224	for (unsigned i = 0; i < RT_ELEMENTS(pVCpu->pgm.s.apGstPaePDsR3); i++)
1225	{
1226	pPGM->apGstPaePDsR3[i] = NULL;
1227	#ifndef VBOX_WITH_2X_4GB_ADDR_SPACE
1228	pPGM->apGstPaePDsR0[i] = NIL_RTR0PTR;
1229	#endif
1230	pPGM->apGstPaePDsRC[i] = NIL_RTRCPTR;
1231	pPGM->aGCPhysGstPaePDs[i] = NIL_RTGCPHYS;
1232	pPGM->aGCPhysGstPaePDsMonitored[i] = NIL_RTGCPHYS;
1233	}
1234
1235	pPGM->fA20Enabled = true;
1236	}
1237
1238	pVM->pgm.s.enmHostMode = SUPPAGINGMODE_INVALID;
1239	pVM->pgm.s.GCPhys4MBPSEMask = RT_BIT_64(32) - 1; /* default; checked later */
1240	pVM->pgm.s.GCPtrPrevRamRangeMapping = MM_HYPER_AREA_ADDRESS;
1241
1242	rc = CFGMR3QueryBoolDef(CFGMR3GetRoot(pVM), "RamPreAlloc", &pVM->pgm.s.fRamPreAlloc,
1243	#ifdef VBOX_WITH_PREALLOC_RAM_BY_DEFAULT
1244	true
1245	#else
1246	false
1247	#endif
1248	);
1249	AssertLogRelRCReturn(rc, rc);
1250
1251	#ifdef PGM_WITH_LARGE_ADDRESS_SPACE_ON_32_BIT_HOST
1252	rc = CFGMR3QueryU32Def(pCfgPGM, "MaxRing3Chunks", &pVM->pgm.s.ChunkR3Map.cMax, _1G / GMM_CHUNK_SIZE);
1253	#else
1254	rc = CFGMR3QueryU32Def(pCfgPGM, "MaxRing3Chunks", &pVM->pgm.s.ChunkR3Map.cMax, UINT32_MAX);
1255	#endif
1256	AssertLogRelRCReturn(rc, rc);
1257	for (uint32_t i = 0; i < RT_ELEMENTS(pVM->pgm.s.ChunkR3Map.Tlb.aEntries); i++)
1258	pVM->pgm.s.ChunkR3Map.Tlb.aEntries[i].idChunk = NIL_GMM_CHUNKID;
1259
1260	/*
1261	* Get the configured RAM size - to estimate saved state size.
1262	*/
1263	uint64_t cbRam;
1264	rc = CFGMR3QueryU64(CFGMR3GetRoot(pVM), "RamSize", &cbRam);
1265	if (rc == VERR_CFGM_VALUE_NOT_FOUND)
1266	cbRam = 0;
1267	else if (RT_SUCCESS(rc))
1268	{
1269	if (cbRam < PAGE_SIZE)
1270	cbRam = 0;
1271	cbRam = RT_ALIGN_64(cbRam, PAGE_SIZE);
1272	}
1273	else
1274	{
1275	AssertMsgFailed(("Configuration error: Failed to query integer \"RamSize\", rc=%Rrc.\n", rc));
1276	return rc;
1277	}
1278
1279	/*
1280	* Check for PCI pass-through.
1281	*/
1282	rc = CFGMR3QueryBoolDef(pCfgPGM, "PciPassThrough", &pVM->pgm.s.fPciPassthrough, false);
1283	AssertMsgRCReturn(rc, ("Configuration error: Failed to query integer \"PciPassThrough\", rc=%Rrc.\n", rc), rc);
1284	AssertLogRelReturn(!pVM->pgm.s.fPciPassthrough \|\| pVM->pgm.s.fRamPreAlloc, VERR_INVALID_PARAMETER);
1285
1286	#ifdef VBOX_WITH_STATISTICS
1287	/*
1288	* Allocate memory for the statistics before someone tries to use them.
1289	*/
1290	size_t cbTotalStats = RT_ALIGN_Z(sizeof(PGMSTATS), 64) + RT_ALIGN_Z(sizeof(PGMCPUSTATS), 64) * pVM->cCpus;
1291	void *pv;
1292	rc = MMHyperAlloc(pVM, RT_ALIGN_Z(cbTotalStats, PAGE_SIZE), PAGE_SIZE, MM_TAG_PGM, &pv);
1293	AssertRCReturn(rc, rc);
1294
1295	pVM->pgm.s.pStatsR3 = (PGMSTATS *)pv;
1296	pVM->pgm.s.pStatsR0 = MMHyperCCToR0(pVM, pv);
1297	pVM->pgm.s.pStatsRC = MMHyperCCToRC(pVM, pv);
1298	pv = (uint8_t *)pv + RT_ALIGN_Z(sizeof(PGMSTATS), 64);
1299
1300	for (VMCPUID iCpu = 0; iCpu < pVM->cCpus; iCpu++)
1301	{
1302	pVM->aCpus[iCpu].pgm.s.pStatsR3 = (PGMCPUSTATS *)pv;
1303	pVM->aCpus[iCpu].pgm.s.pStatsR0 = MMHyperCCToR0(pVM, pv);
1304	pVM->aCpus[iCpu].pgm.s.pStatsRC = MMHyperCCToRC(pVM, pv);
1305
1306	pv = (uint8_t *)pv + RT_ALIGN_Z(sizeof(PGMCPUSTATS), 64);
1307	}
1308	#endif /* VBOX_WITH_STATISTICS */
1309
1310	/*
1311	* Register callbacks, string formatters and the saved state data unit.
1312	*/
1313	#ifdef VBOX_STRICT
1314	VMR3AtStateRegister(pVM, pgmR3ResetNoMorePhysWritesFlag, NULL);
1315	#endif
1316	PGMRegisterStringFormatTypes();
1317
1318	rc = pgmR3InitSavedState(pVM, cbRam);
1319	if (RT_FAILURE(rc))
1320	return rc;
1321
1322	/*
1323	* Initialize the PGM critical section and flush the phys TLBs
1324	*/
1325	rc = PDMR3CritSectInit(pVM, &pVM->pgm.s.CritSect, RT_SRC_POS, "PGM");
1326	AssertRCReturn(rc, rc);
1327
1328	PGMR3PhysChunkInvalidateTLB(pVM);
1329	PGMPhysInvalidatePageMapTLB(pVM);
1330
1331	/*
1332	* For the time being we sport a full set of handy pages in addition to the base
1333	* memory to simplify things.
1334	*/
1335	rc = MMR3ReserveHandyPages(pVM, RT_ELEMENTS(pVM->pgm.s.aHandyPages)); /** @todo this should be changed to PGM_HANDY_PAGES_MIN but this needs proper testing... */
1336	AssertRCReturn(rc, rc);
1337
1338	/*
1339	* Trees
1340	*/
1341	rc = MMHyperAlloc(pVM, sizeof(PGMTREES), 0, MM_TAG_PGM, (void **)&pVM->pgm.s.pTreesR3);
1342	if (RT_SUCCESS(rc))
1343	{
1344	pVM->pgm.s.pTreesR0 = MMHyperR3ToR0(pVM, pVM->pgm.s.pTreesR3);
1345	pVM->pgm.s.pTreesRC = MMHyperR3ToRC(pVM, pVM->pgm.s.pTreesR3);
1346
1347	/*
1348	* Allocate the zero page.
1349	*/
1350	rc = MMHyperAlloc(pVM, PAGE_SIZE, PAGE_SIZE, MM_TAG_PGM, &pVM->pgm.s.pvZeroPgR3);
1351	}
1352	if (RT_SUCCESS(rc))
1353	{
1354	pVM->pgm.s.pvZeroPgRC = MMHyperR3ToRC(pVM, pVM->pgm.s.pvZeroPgR3);
1355	pVM->pgm.s.pvZeroPgR0 = MMHyperR3ToR0(pVM, pVM->pgm.s.pvZeroPgR3);
1356	pVM->pgm.s.HCPhysZeroPg = MMR3HyperHCVirt2HCPhys(pVM, pVM->pgm.s.pvZeroPgR3);
1357	AssertRelease(pVM->pgm.s.HCPhysZeroPg != NIL_RTHCPHYS);
1358
1359	/*
1360	* Allocate the invalid MMIO page.
1361	* (The invalid bits in HCPhysInvMmioPg are set later on init complete.)
1362	*/
1363	rc = MMHyperAlloc(pVM, PAGE_SIZE, PAGE_SIZE, MM_TAG_PGM, &pVM->pgm.s.pvMmioPgR3);
1364	}
1365	if (RT_SUCCESS(rc))
1366	{
1367	ASMMemFill32(pVM->pgm.s.pvMmioPgR3, PAGE_SIZE, 0xfeedface);
1368	pVM->pgm.s.HCPhysMmioPg = MMR3HyperHCVirt2HCPhys(pVM, pVM->pgm.s.pvMmioPgR3);
1369	AssertRelease(pVM->pgm.s.HCPhysMmioPg != NIL_RTHCPHYS);
1370	pVM->pgm.s.HCPhysInvMmioPg = pVM->pgm.s.HCPhysMmioPg;
1371
1372	/*
1373	* Init the paging.
1374	*/
1375	rc = pgmR3InitPaging(pVM);
1376	}
1377	if (RT_SUCCESS(rc))
1378	{
1379	/*
1380	* Init the page pool.
1381	*/
1382	rc = pgmR3PoolInit(pVM);
1383	}
1384	if (RT_SUCCESS(rc))
1385	{
1386	for (VMCPUID i = 0; i < pVM->cCpus; i++)
1387	{
1388	PVMCPU pVCpu = &pVM->aCpus[i];
1389	rc = PGMR3ChangeMode(pVM, pVCpu, PGMMODE_REAL);
1390	if (RT_FAILURE(rc))
1391	break;
1392	}
1393	}
1394
1395	if (RT_SUCCESS(rc))
1396	{
1397	/*
1398	* Info & statistics
1399	*/
1400	DBGFR3InfoRegisterInternal(pVM, "mode",
1401	"Shows the current paging mode. "
1402	"Recognizes 'all', 'guest', 'shadow' and 'host' as arguments, defaulting to 'all' if nothing is given.",
1403	pgmR3InfoMode);
1404	DBGFR3InfoRegisterInternal(pVM, "pgmcr3",
1405	"Dumps all the entries in the top level paging table. No arguments.",
1406	pgmR3InfoCr3);
1407	DBGFR3InfoRegisterInternal(pVM, "phys",
1408	"Dumps all the physical address ranges. No arguments.",
1409	pgmR3PhysInfo);
1410	DBGFR3InfoRegisterInternal(pVM, "handlers",
1411	"Dumps physical, virtual and hyper virtual handlers. "
1412	"Pass 'phys', 'virt', 'hyper' as argument if only one kind is wanted."
1413	"Add 'nost' if the statistics are unwanted, use together with 'all' or explicit selection.",
1414	pgmR3InfoHandlers);
1415	DBGFR3InfoRegisterInternal(pVM, "mappings",
1416	"Dumps guest mappings.",
1417	pgmR3MapInfo);
1418
1419	pgmR3InitStats(pVM);
1420
1421	#ifdef VBOX_WITH_DEBUGGER
1422	/*
1423	* Debugger commands.
1424	*/
1425	static bool s_fRegisteredCmds = false;
1426	if (!s_fRegisteredCmds)
1427	{
1428	int rc2 = DBGCRegisterCommands(&g_aCmds[0], RT_ELEMENTS(g_aCmds));
1429	if (RT_SUCCESS(rc2))
1430	s_fRegisteredCmds = true;
1431	}
1432	#endif
1433	return VINF_SUCCESS;
1434	}
1435
1436	/* Almost no cleanup necessary, MM frees all memory. */
1437	PDMR3CritSectDelete(&pVM->pgm.s.CritSect);
1438
1439	return rc;
1440	}
1441
1442
1443	/**
1444	* Init paging.
1445	*
1446	* Since we need to check what mode the host is operating in before we can choose
1447	* the right paging functions for the host we have to delay this until R0 has
1448	* been initialized.
1449	*
1450	* @returns VBox status code.
1451	* @param pVM VM handle.
1452	*/
1453	static int pgmR3InitPaging(PVM pVM)
1454	{
1455	/*
1456	* Force a recalculation of modes and switcher so everyone gets notified.
1457	*/
1458	for (VMCPUID i = 0; i < pVM->cCpus; i++)
1459	{
1460	PVMCPU pVCpu = &pVM->aCpus[i];
1461
1462	pVCpu->pgm.s.enmShadowMode = PGMMODE_INVALID;
1463	pVCpu->pgm.s.enmGuestMode = PGMMODE_INVALID;
1464	}
1465
1466	pVM->pgm.s.enmHostMode = SUPPAGINGMODE_INVALID;
1467
1468	/*
1469	* Allocate static mapping space for whatever the cr3 register
1470	* points to and in the case of PAE mode to the 4 PDs.
1471	*/
1472	int rc = MMR3HyperReserve(pVM, PAGE_SIZE * 5, "CR3 mapping", &pVM->pgm.s.GCPtrCR3Mapping);
1473	if (RT_FAILURE(rc))
1474	{
1475	AssertMsgFailed(("Failed to reserve two pages for cr mapping in HMA, rc=%Rrc\n", rc));
1476	return rc;
1477	}
1478	MMR3HyperReserve(pVM, PAGE_SIZE, "fence", NULL);
1479
1480	/*
1481	* Allocate pages for the three possible intermediate contexts
1482	* (AMD64, PAE and plain 32-Bit). We maintain all three contexts
1483	* for the sake of simplicity. The AMD64 uses the PAE for the
1484	* lower levels, making the total number of pages 11 (3 + 7 + 1).
1485	*
1486	* We assume that two page tables will be enought for the core code
1487	* mappings (HC virtual and identity).
1488	*/
1489	pVM->pgm.s.pInterPD = (PX86PD)MMR3PageAllocLow(pVM); AssertReturn(pVM->pgm.s.pInterPD, VERR_NO_PAGE_MEMORY);
1490	pVM->pgm.s.apInterPTs[0] = (PX86PT)MMR3PageAllocLow(pVM); AssertReturn(pVM->pgm.s.apInterPTs[0], VERR_NO_PAGE_MEMORY);
1491	pVM->pgm.s.apInterPTs[1] = (PX86PT)MMR3PageAllocLow(pVM); AssertReturn(pVM->pgm.s.apInterPTs[1], VERR_NO_PAGE_MEMORY);
1492	pVM->pgm.s.apInterPaePTs[0] = (PX86PTPAE)MMR3PageAlloc(pVM); AssertReturn(pVM->pgm.s.apInterPaePTs[0], VERR_NO_PAGE_MEMORY);
1493	pVM->pgm.s.apInterPaePTs[1] = (PX86PTPAE)MMR3PageAlloc(pVM); AssertReturn(pVM->pgm.s.apInterPaePTs[1], VERR_NO_PAGE_MEMORY);
1494	pVM->pgm.s.apInterPaePDs[0] = (PX86PDPAE)MMR3PageAlloc(pVM); AssertReturn(pVM->pgm.s.apInterPaePDs[0], VERR_NO_PAGE_MEMORY);
1495	pVM->pgm.s.apInterPaePDs[1] = (PX86PDPAE)MMR3PageAlloc(pVM); AssertReturn(pVM->pgm.s.apInterPaePDs[1], VERR_NO_PAGE_MEMORY);
1496	pVM->pgm.s.apInterPaePDs[2] = (PX86PDPAE)MMR3PageAlloc(pVM); AssertReturn(pVM->pgm.s.apInterPaePDs[2], VERR_NO_PAGE_MEMORY);
1497	pVM->pgm.s.apInterPaePDs[3] = (PX86PDPAE)MMR3PageAlloc(pVM); AssertReturn(pVM->pgm.s.apInterPaePDs[3], VERR_NO_PAGE_MEMORY);
1498	pVM->pgm.s.pInterPaePDPT = (PX86PDPT)MMR3PageAllocLow(pVM); AssertReturn(pVM->pgm.s.pInterPaePDPT, VERR_NO_PAGE_MEMORY);
1499	pVM->pgm.s.pInterPaePDPT64 = (PX86PDPT)MMR3PageAllocLow(pVM); AssertReturn(pVM->pgm.s.pInterPaePDPT64, VERR_NO_PAGE_MEMORY);
1500	pVM->pgm.s.pInterPaePML4 = (PX86PML4)MMR3PageAllocLow(pVM); AssertReturn(pVM->pgm.s.pInterPaePML4, VERR_NO_PAGE_MEMORY);
1501
1502	pVM->pgm.s.HCPhysInterPD = MMPage2Phys(pVM, pVM->pgm.s.pInterPD);
1503	AssertRelease(pVM->pgm.s.HCPhysInterPD != NIL_RTHCPHYS && !(pVM->pgm.s.HCPhysInterPD & PAGE_OFFSET_MASK));
1504	pVM->pgm.s.HCPhysInterPaePDPT = MMPage2Phys(pVM, pVM->pgm.s.pInterPaePDPT);
1505	AssertRelease(pVM->pgm.s.HCPhysInterPaePDPT != NIL_RTHCPHYS && !(pVM->pgm.s.HCPhysInterPaePDPT & PAGE_OFFSET_MASK));
1506	pVM->pgm.s.HCPhysInterPaePML4 = MMPage2Phys(pVM, pVM->pgm.s.pInterPaePML4);
1507	AssertRelease(pVM->pgm.s.HCPhysInterPaePML4 != NIL_RTHCPHYS && !(pVM->pgm.s.HCPhysInterPaePML4 & PAGE_OFFSET_MASK) && pVM->pgm.s.HCPhysInterPaePML4 < 0xffffffff);
1508
1509	/*
1510	* Initialize the pages, setting up the PML4 and PDPT for repetitive 4GB action.
1511	*/
1512	ASMMemZeroPage(pVM->pgm.s.pInterPD);
1513	ASMMemZeroPage(pVM->pgm.s.apInterPTs[0]);
1514	ASMMemZeroPage(pVM->pgm.s.apInterPTs[1]);
1515
1516	ASMMemZeroPage(pVM->pgm.s.apInterPaePTs[0]);
1517	ASMMemZeroPage(pVM->pgm.s.apInterPaePTs[1]);
1518
1519	ASMMemZeroPage(pVM->pgm.s.pInterPaePDPT);
1520	for (unsigned i = 0; i < RT_ELEMENTS(pVM->pgm.s.apInterPaePDs); i++)
1521	{
1522	ASMMemZeroPage(pVM->pgm.s.apInterPaePDs[i]);
1523	pVM->pgm.s.pInterPaePDPT->a[i].u = X86_PDPE_P \| PGM_PLXFLAGS_PERMANENT
1524	\| MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[i]);
1525	}
1526
1527	for (unsigned i = 0; i < RT_ELEMENTS(pVM->pgm.s.pInterPaePDPT64->a); i++)
1528	{
1529	const unsigned iPD = i % RT_ELEMENTS(pVM->pgm.s.apInterPaePDs);
1530	pVM->pgm.s.pInterPaePDPT64->a[i].u = X86_PDPE_P \| X86_PDPE_RW \| X86_PDPE_US \| X86_PDPE_A \| PGM_PLXFLAGS_PERMANENT
1531	\| MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[iPD]);
1532	}
1533
1534	RTHCPHYS HCPhysInterPaePDPT64 = MMPage2Phys(pVM, pVM->pgm.s.pInterPaePDPT64);
1535	for (unsigned i = 0; i < RT_ELEMENTS(pVM->pgm.s.pInterPaePML4->a); i++)
1536	pVM->pgm.s.pInterPaePML4->a[i].u = X86_PML4E_P \| X86_PML4E_RW \| X86_PML4E_US \| X86_PML4E_A \| PGM_PLXFLAGS_PERMANENT
1537	\| HCPhysInterPaePDPT64;
1538
1539	/*
1540	* Initialize paging workers and mode from current host mode
1541	* and the guest running in real mode.
1542	*/
1543	pVM->pgm.s.enmHostMode = SUPR3GetPagingMode();
1544	switch (pVM->pgm.s.enmHostMode)
1545	{
1546	case SUPPAGINGMODE_32_BIT:
1547	case SUPPAGINGMODE_32_BIT_GLOBAL:
1548	case SUPPAGINGMODE_PAE:
1549	case SUPPAGINGMODE_PAE_GLOBAL:
1550	case SUPPAGINGMODE_PAE_NX:
1551	case SUPPAGINGMODE_PAE_GLOBAL_NX:
1552	break;
1553
1554	case SUPPAGINGMODE_AMD64:
1555	case SUPPAGINGMODE_AMD64_GLOBAL:
1556	case SUPPAGINGMODE_AMD64_NX:
1557	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
1558	#ifndef VBOX_WITH_HYBRID_32BIT_KERNEL
1559	if (ARCH_BITS != 64)
1560	{
1561	AssertMsgFailed(("Host mode %d (64-bit) is not supported by non-64bit builds\n", pVM->pgm.s.enmHostMode));
1562	LogRel(("Host mode %d (64-bit) is not supported by non-64bit builds\n", pVM->pgm.s.enmHostMode));
1563	return VERR_PGM_UNSUPPORTED_HOST_PAGING_MODE;
1564	}
1565	#endif
1566	break;
1567	default:
1568	AssertMsgFailed(("Host mode %d is not supported\n", pVM->pgm.s.enmHostMode));
1569	return VERR_PGM_UNSUPPORTED_HOST_PAGING_MODE;
1570	}
1571	rc = pgmR3ModeDataInit(pVM, false /* don't resolve GC and R0 syms yet */);
1572	if (RT_SUCCESS(rc))
1573	{
1574	LogFlow(("pgmR3InitPaging: returns successfully\n"));
1575	#if HC_ARCH_BITS == 64
1576	LogRel(("Debug: HCPhysInterPD=%RHp HCPhysInterPaePDPT=%RHp HCPhysInterPaePML4=%RHp\n",
1577	pVM->pgm.s.HCPhysInterPD, pVM->pgm.s.HCPhysInterPaePDPT, pVM->pgm.s.HCPhysInterPaePML4));
1578	LogRel(("Debug: apInterPTs={%RHp,%RHp} apInterPaePTs={%RHp,%RHp} apInterPaePDs={%RHp,%RHp,%RHp,%RHp} pInterPaePDPT64=%RHp\n",
1579	MMPage2Phys(pVM, pVM->pgm.s.apInterPTs[0]), MMPage2Phys(pVM, pVM->pgm.s.apInterPTs[1]),
1580	MMPage2Phys(pVM, pVM->pgm.s.apInterPaePTs[0]), MMPage2Phys(pVM, pVM->pgm.s.apInterPaePTs[1]),
1581	MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[0]), MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[1]), MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[2]), MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[3]),
1582	MMPage2Phys(pVM, pVM->pgm.s.pInterPaePDPT64)));
1583	#endif
1584	return VINF_SUCCESS;
1585	}
1586
1587	LogFlow(("pgmR3InitPaging: returns %Rrc\n", rc));
1588	return rc;
1589	}
1590
1591
1592	/**
1593	* Init statistics
1594	* @returns VBox status code.
1595	*/
1596	static int pgmR3InitStats(PVM pVM)
1597	{
1598	PPGM pPGM = &pVM->pgm.s;
1599	int rc;
1600
1601	/*
1602	* Release statistics.
1603	*/
1604	/* Common - misc variables */
1605	STAM_REL_REG(pVM, &pPGM->cAllPages, STAMTYPE_U32, "/PGM/Page/cAllPages", STAMUNIT_COUNT, "The total number of pages.");
1606	STAM_REL_REG(pVM, &pPGM->cPrivatePages, STAMTYPE_U32, "/PGM/Page/cPrivatePages", STAMUNIT_COUNT, "The number of private pages.");
1607	STAM_REL_REG(pVM, &pPGM->cSharedPages, STAMTYPE_U32, "/PGM/Page/cSharedPages", STAMUNIT_COUNT, "The number of shared pages.");
1608	STAM_REL_REG(pVM, &pPGM->cReusedSharedPages, STAMTYPE_U32, "/PGM/Page/cReusedSharedPages", STAMUNIT_COUNT, "The number of reused shared pages.");
1609	STAM_REL_REG(pVM, &pPGM->cZeroPages, STAMTYPE_U32, "/PGM/Page/cZeroPages", STAMUNIT_COUNT, "The number of zero backed pages.");
1610	STAM_REL_REG(pVM, &pPGM->cPureMmioPages, STAMTYPE_U32, "/PGM/Page/cPureMmioPages", STAMUNIT_COUNT, "The number of pure MMIO pages.");
1611	STAM_REL_REG(pVM, &pPGM->cMonitoredPages, STAMTYPE_U32, "/PGM/Page/cMonitoredPages", STAMUNIT_COUNT, "The number of write monitored pages.");
1612	STAM_REL_REG(pVM, &pPGM->cWrittenToPages, STAMTYPE_U32, "/PGM/Page/cWrittenToPages", STAMUNIT_COUNT, "The number of previously write monitored pages that have been written to.");
1613	STAM_REL_REG(pVM, &pPGM->cWriteLockedPages, STAMTYPE_U32, "/PGM/Page/cWriteLockedPages", STAMUNIT_COUNT, "The number of write(/read) locked pages.");
1614	STAM_REL_REG(pVM, &pPGM->cReadLockedPages, STAMTYPE_U32, "/PGM/Page/cReadLockedPages", STAMUNIT_COUNT, "The number of read (only) locked pages.");
1615	STAM_REL_REG(pVM, &pPGM->cBalloonedPages, STAMTYPE_U32, "/PGM/Page/cBalloonedPages", STAMUNIT_COUNT, "The number of ballooned pages.");
1616	STAM_REL_REG(pVM, &pPGM->cHandyPages, STAMTYPE_U32, "/PGM/Page/cHandyPages", STAMUNIT_COUNT, "The number of handy pages (not included in cAllPages).");
1617	STAM_REL_REG(pVM, &pPGM->cLargePages, STAMTYPE_U32, "/PGM/Page/cLargePages", STAMUNIT_COUNT, "The number of large pages allocated (includes disabled).");
1618	STAM_REL_REG(pVM, &pPGM->cLargePagesDisabled, STAMTYPE_U32, "/PGM/Page/cLargePagesDisabled", STAMUNIT_COUNT, "The number of disabled large pages.");
1619	STAM_REL_REG(pVM, &pPGM->cRelocations, STAMTYPE_COUNTER, "/PGM/cRelocations", STAMUNIT_OCCURENCES,"Number of hypervisor relocations.");
1620	STAM_REL_REG(pVM, &pPGM->ChunkR3Map.c, STAMTYPE_U32, "/PGM/ChunkR3Map/c", STAMUNIT_COUNT, "Number of mapped chunks.");
1621	STAM_REL_REG(pVM, &pPGM->ChunkR3Map.cMax, STAMTYPE_U32, "/PGM/ChunkR3Map/cMax", STAMUNIT_COUNT, "Maximum number of mapped chunks.");
1622	STAM_REL_REG(pVM, &pPGM->cMappedChunks, STAMTYPE_U32, "/PGM/ChunkR3Map/Mapped", STAMUNIT_COUNT, "Number of times we mapped a chunk.");
1623	STAM_REL_REG(pVM, &pPGM->cUnmappedChunks, STAMTYPE_U32, "/PGM/ChunkR3Map/Unmapped", STAMUNIT_COUNT, "Number of times we unmapped a chunk.");
1624
1625	STAM_REL_REG(pVM, &pPGM->StatLargePageReused, STAMTYPE_COUNTER, "/PGM/LargePage/Reused", STAMUNIT_OCCURENCES, "The number of times we've reused a large page.");
1626	STAM_REL_REG(pVM, &pPGM->StatLargePageRefused, STAMTYPE_COUNTER, "/PGM/LargePage/Refused", STAMUNIT_OCCURENCES, "The number of times we couldn't use a large page.");
1627	STAM_REL_REG(pVM, &pPGM->StatLargePageRecheck, STAMTYPE_COUNTER, "/PGM/LargePage/Recheck", STAMUNIT_OCCURENCES, "The number of times we've rechecked a disabled large page.");
1628
1629	/* Live save */
1630	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.fActive, STAMTYPE_U8, "/PGM/LiveSave/fActive", STAMUNIT_COUNT, "Active or not.");
1631	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.cIgnoredPages, STAMTYPE_U32, "/PGM/LiveSave/cIgnoredPages", STAMUNIT_COUNT, "The number of ignored pages in the RAM ranges (i.e. MMIO, MMIO2 and ROM).");
1632	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.cDirtyPagesLong, STAMTYPE_U32, "/PGM/LiveSave/cDirtyPagesLong", STAMUNIT_COUNT, "Longer term dirty page average.");
1633	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.cDirtyPagesShort, STAMTYPE_U32, "/PGM/LiveSave/cDirtyPagesShort", STAMUNIT_COUNT, "Short term dirty page average.");
1634	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.cPagesPerSecond, STAMTYPE_U32, "/PGM/LiveSave/cPagesPerSecond", STAMUNIT_COUNT, "Pages per second.");
1635	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.cSavedPages, STAMTYPE_U64, "/PGM/LiveSave/cSavedPages", STAMUNIT_COUNT, "The total number of saved pages.");
1636	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Ram.cReadyPages, STAMTYPE_U32, "/PGM/LiveSave/Ram/cReadPages", STAMUNIT_COUNT, "RAM: Ready pages.");
1637	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Ram.cDirtyPages, STAMTYPE_U32, "/PGM/LiveSave/Ram/cDirtyPages", STAMUNIT_COUNT, "RAM: Dirty pages.");
1638	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Ram.cZeroPages, STAMTYPE_U32, "/PGM/LiveSave/Ram/cZeroPages", STAMUNIT_COUNT, "RAM: Ready zero pages.");
1639	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Ram.cMonitoredPages, STAMTYPE_U32, "/PGM/LiveSave/Ram/cMonitoredPages", STAMUNIT_COUNT, "RAM: Write monitored pages.");
1640	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Rom.cReadyPages, STAMTYPE_U32, "/PGM/LiveSave/Rom/cReadPages", STAMUNIT_COUNT, "ROM: Ready pages.");
1641	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Rom.cDirtyPages, STAMTYPE_U32, "/PGM/LiveSave/Rom/cDirtyPages", STAMUNIT_COUNT, "ROM: Dirty pages.");
1642	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Rom.cZeroPages, STAMTYPE_U32, "/PGM/LiveSave/Rom/cZeroPages", STAMUNIT_COUNT, "ROM: Ready zero pages.");
1643	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Rom.cMonitoredPages, STAMTYPE_U32, "/PGM/LiveSave/Rom/cMonitoredPages", STAMUNIT_COUNT, "ROM: Write monitored pages.");
1644	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Mmio2.cReadyPages, STAMTYPE_U32, "/PGM/LiveSave/Mmio2/cReadPages", STAMUNIT_COUNT, "MMIO2: Ready pages.");
1645	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Mmio2.cDirtyPages, STAMTYPE_U32, "/PGM/LiveSave/Mmio2/cDirtyPages", STAMUNIT_COUNT, "MMIO2: Dirty pages.");
1646	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Mmio2.cZeroPages, STAMTYPE_U32, "/PGM/LiveSave/Mmio2/cZeroPages", STAMUNIT_COUNT, "MMIO2: Ready zero pages.");
1647	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Mmio2.cMonitoredPages,STAMTYPE_U32, "/PGM/LiveSave/Mmio2/cMonitoredPages",STAMUNIT_COUNT, "MMIO2: Write monitored pages.");
1648
1649	#ifdef VBOX_WITH_STATISTICS
1650
1651	# define PGM_REG_COUNTER(a, b, c) \
1652	rc = STAMR3RegisterF(pVM, a, STAMTYPE_COUNTER, STAMVISIBILITY_ALWAYS, STAMUNIT_OCCURENCES, c, b); \
1653	AssertRC(rc);
1654
1655	# define PGM_REG_COUNTER_BYTES(a, b, c) \
1656	rc = STAMR3RegisterF(pVM, a, STAMTYPE_COUNTER, STAMVISIBILITY_ALWAYS, STAMUNIT_BYTES, c, b); \
1657	AssertRC(rc);
1658
1659	# define PGM_REG_PROFILE(a, b, c) \
1660	rc = STAMR3RegisterF(pVM, a, STAMTYPE_PROFILE, STAMVISIBILITY_ALWAYS, STAMUNIT_TICKS_PER_CALL, c, b); \
1661	AssertRC(rc);
1662
1663	PGMSTATS *pStats = pVM->pgm.s.pStatsR3;
1664
1665	PGM_REG_PROFILE(&pStats->StatAllocLargePage, "/PGM/LargePage/Prof/Alloc", "Time spent by the host OS for large page allocation.");
1666	PGM_REG_PROFILE(&pStats->StatClearLargePage, "/PGM/LargePage/Prof/Clear", "Time spent clearing the newly allocated large pages.");
1667	PGM_REG_COUNTER(&pStats->StatLargePageOverflow, "/PGM/LargePage/Overflow", "The number of times allocating a large page took too long.");
1668	PGM_REG_PROFILE(&pStats->StatR3IsValidLargePage, "/PGM/LargePage/Prof/R3/IsValid", "pgmPhysIsValidLargePage profiling - R3.");
1669	PGM_REG_PROFILE(&pStats->StatRZIsValidLargePage, "/PGM/LargePage/Prof/RZ/IsValid", "pgmPhysIsValidLargePage profiling - RZ.");
1670
1671	PGM_REG_COUNTER(&pStats->StatR3DetectedConflicts, "/PGM/R3/DetectedConflicts", "The number of times PGMR3CheckMappingConflicts() detected a conflict.");
1672	PGM_REG_PROFILE(&pStats->StatR3ResolveConflict, "/PGM/R3/ResolveConflict", "pgmR3SyncPTResolveConflict() profiling (includes the entire relocation).");
1673	PGM_REG_COUNTER(&pStats->StatR3PhysRead, "/PGM/R3/Phys/Read", "The number of times PGMPhysRead was called.");
1674	PGM_REG_COUNTER_BYTES(&pStats->StatR3PhysReadBytes, "/PGM/R3/Phys/Read/Bytes", "The number of bytes read by PGMPhysRead.");
1675	PGM_REG_COUNTER(&pStats->StatR3PhysWrite, "/PGM/R3/Phys/Write", "The number of times PGMPhysWrite was called.");
1676	PGM_REG_COUNTER_BYTES(&pStats->StatR3PhysWriteBytes, "/PGM/R3/Phys/Write/Bytes", "The number of bytes written by PGMPhysWrite.");
1677	PGM_REG_COUNTER(&pStats->StatR3PhysSimpleRead, "/PGM/R3/Phys/Simple/Read", "The number of times PGMPhysSimpleReadGCPtr was called.");
1678	PGM_REG_COUNTER_BYTES(&pStats->StatR3PhysSimpleReadBytes, "/PGM/R3/Phys/Simple/Read/Bytes", "The number of bytes read by PGMPhysSimpleReadGCPtr.");
1679	PGM_REG_COUNTER(&pStats->StatR3PhysSimpleWrite, "/PGM/R3/Phys/Simple/Write", "The number of times PGMPhysSimpleWriteGCPtr was called.");
1680	PGM_REG_COUNTER_BYTES(&pStats->StatR3PhysSimpleWriteBytes, "/PGM/R3/Phys/Simple/Write/Bytes", "The number of bytes written by PGMPhysSimpleWriteGCPtr.");
1681
1682	PGM_REG_COUNTER(&pStats->StatRZChunkR3MapTlbHits, "/PGM/ChunkR3Map/TlbHitsRZ", "TLB hits.");
1683	PGM_REG_COUNTER(&pStats->StatRZChunkR3MapTlbMisses, "/PGM/ChunkR3Map/TlbMissesRZ", "TLB misses.");
1684	PGM_REG_PROFILE(&pStats->StatChunkAging, "/PGM/ChunkR3Map/Map/Aging", "Chunk aging profiling.");
1685	PGM_REG_PROFILE(&pStats->StatChunkFindCandidate, "/PGM/ChunkR3Map/Map/Find", "Chunk unmap find profiling.");
1686	PGM_REG_PROFILE(&pStats->StatChunkUnmap, "/PGM/ChunkR3Map/Map/Unmap", "Chunk unmap of address space profiling.");
1687	PGM_REG_PROFILE(&pStats->StatChunkMap, "/PGM/ChunkR3Map/Map/Map", "Chunk map of address space profiling.");
1688
1689	PGM_REG_COUNTER(&pStats->StatRZPageMapTlbHits, "/PGM/RZ/Page/MapTlbHits", "TLB hits.");
1690	PGM_REG_COUNTER(&pStats->StatRZPageMapTlbMisses, "/PGM/RZ/Page/MapTlbMisses", "TLB misses.");
1691	PGM_REG_COUNTER(&pStats->StatR3ChunkR3MapTlbHits, "/PGM/ChunkR3Map/TlbHitsR3", "TLB hits.");
1692	PGM_REG_COUNTER(&pStats->StatR3ChunkR3MapTlbMisses, "/PGM/ChunkR3Map/TlbMissesR3", "TLB misses.");
1693	PGM_REG_COUNTER(&pStats->StatR3PageMapTlbHits, "/PGM/R3/Page/MapTlbHits", "TLB hits.");
1694	PGM_REG_COUNTER(&pStats->StatR3PageMapTlbMisses, "/PGM/R3/Page/MapTlbMisses", "TLB misses.");
1695	PGM_REG_COUNTER(&pStats->StatPageMapTlbFlushes, "/PGM/R3/Page/MapTlbFlushes", "TLB flushes (all contexts).");
1696	PGM_REG_COUNTER(&pStats->StatPageMapTlbFlushEntry, "/PGM/R3/Page/MapTlbFlushEntry", "TLB entry flushes (all contexts).");
1697
1698	PGM_REG_PROFILE(&pStats->StatRZSyncCR3HandlerVirtualUpdate, "/PGM/RZ/SyncCR3/Handlers/VirtualUpdate", "Profiling of the virtual handler updates.");
1699	PGM_REG_PROFILE(&pStats->StatRZSyncCR3HandlerVirtualReset, "/PGM/RZ/SyncCR3/Handlers/VirtualReset", "Profiling of the virtual handler resets.");
1700	PGM_REG_PROFILE(&pStats->StatR3SyncCR3HandlerVirtualUpdate, "/PGM/R3/SyncCR3/Handlers/VirtualUpdate", "Profiling of the virtual handler updates.");
1701	PGM_REG_PROFILE(&pStats->StatR3SyncCR3HandlerVirtualReset, "/PGM/R3/SyncCR3/Handlers/VirtualReset", "Profiling of the virtual handler resets.");
1702
1703	PGM_REG_COUNTER(&pStats->StatRZPhysHandlerReset, "/PGM/RZ/PhysHandlerReset", "The number of times PGMHandlerPhysicalReset is called.");
1704	PGM_REG_COUNTER(&pStats->StatR3PhysHandlerReset, "/PGM/R3/PhysHandlerReset", "The number of times PGMHandlerPhysicalReset is called.");
1705	PGM_REG_COUNTER(&pStats->StatRZPhysHandlerLookupHits, "/PGM/RZ/PhysHandlerLookupHits", "The number of cache hits when looking up physical handlers.");
1706	PGM_REG_COUNTER(&pStats->StatR3PhysHandlerLookupHits, "/PGM/R3/PhysHandlerLookupHits", "The number of cache hits when looking up physical handlers.");
1707	PGM_REG_COUNTER(&pStats->StatRZPhysHandlerLookupMisses, "/PGM/RZ/PhysHandlerLookupMisses", "The number of cache misses when looking up physical handlers.");
1708	PGM_REG_COUNTER(&pStats->StatR3PhysHandlerLookupMisses, "/PGM/R3/PhysHandlerLookupMisses", "The number of cache misses when looking up physical handlers.");
1709	PGM_REG_PROFILE(&pStats->StatRZVirtHandlerSearchByPhys, "/PGM/RZ/VirtHandlerSearchByPhys", "Profiling of pgmHandlerVirtualFindByPhysAddr.");
1710	PGM_REG_PROFILE(&pStats->StatR3VirtHandlerSearchByPhys, "/PGM/R3/VirtHandlerSearchByPhys", "Profiling of pgmHandlerVirtualFindByPhysAddr.");
1711
1712	PGM_REG_COUNTER(&pStats->StatRZPageReplaceShared, "/PGM/RZ/Page/ReplacedShared", "Times a shared page was replaced.");
1713	PGM_REG_COUNTER(&pStats->StatRZPageReplaceZero, "/PGM/RZ/Page/ReplacedZero", "Times the zero page was replaced.");
1714	/// @todo PGM_REG_COUNTER(&pStats->StatRZPageHandyAllocs, "/PGM/RZ/Page/HandyAllocs", "Number of times we've allocated more handy pages.");
1715	PGM_REG_COUNTER(&pStats->StatR3PageReplaceShared, "/PGM/R3/Page/ReplacedShared", "Times a shared page was replaced.");
1716	PGM_REG_COUNTER(&pStats->StatR3PageReplaceZero, "/PGM/R3/Page/ReplacedZero", "Times the zero page was replaced.");
1717	/// @todo PGM_REG_COUNTER(&pStats->StatR3PageHandyAllocs, "/PGM/R3/Page/HandyAllocs", "Number of times we've allocated more handy pages.");
1718
1719	PGM_REG_COUNTER(&pStats->StatRZPhysRead, "/PGM/RZ/Phys/Read", "The number of times PGMPhysRead was called.");
1720	PGM_REG_COUNTER_BYTES(&pStats->StatRZPhysReadBytes, "/PGM/RZ/Phys/Read/Bytes", "The number of bytes read by PGMPhysRead.");
1721	PGM_REG_COUNTER(&pStats->StatRZPhysWrite, "/PGM/RZ/Phys/Write", "The number of times PGMPhysWrite was called.");
1722	PGM_REG_COUNTER_BYTES(&pStats->StatRZPhysWriteBytes, "/PGM/RZ/Phys/Write/Bytes", "The number of bytes written by PGMPhysWrite.");
1723	PGM_REG_COUNTER(&pStats->StatRZPhysSimpleRead, "/PGM/RZ/Phys/Simple/Read", "The number of times PGMPhysSimpleReadGCPtr was called.");
1724	PGM_REG_COUNTER_BYTES(&pStats->StatRZPhysSimpleReadBytes, "/PGM/RZ/Phys/Simple/Read/Bytes", "The number of bytes read by PGMPhysSimpleReadGCPtr.");
1725	PGM_REG_COUNTER(&pStats->StatRZPhysSimpleWrite, "/PGM/RZ/Phys/Simple/Write", "The number of times PGMPhysSimpleWriteGCPtr was called.");
1726	PGM_REG_COUNTER_BYTES(&pStats->StatRZPhysSimpleWriteBytes, "/PGM/RZ/Phys/Simple/Write/Bytes", "The number of bytes written by PGMPhysSimpleWriteGCPtr.");
1727
1728	/* GC only: */
1729	PGM_REG_COUNTER(&pStats->StatRCInvlPgConflict, "/PGM/RC/InvlPgConflict", "Number of times PGMInvalidatePage() detected a mapping conflict.");
1730	PGM_REG_COUNTER(&pStats->StatRCInvlPgSyncMonCR3, "/PGM/RC/InvlPgSyncMonitorCR3", "Number of times PGMInvalidatePage() ran into PGM_SYNC_MONITOR_CR3.");
1731
1732	PGM_REG_COUNTER(&pStats->StatRCPhysRead, "/PGM/RC/Phys/Read", "The number of times PGMPhysRead was called.");
1733	PGM_REG_COUNTER_BYTES(&pStats->StatRCPhysReadBytes, "/PGM/RC/Phys/Read/Bytes", "The number of bytes read by PGMPhysRead.");
1734	PGM_REG_COUNTER(&pStats->StatRCPhysWrite, "/PGM/RC/Phys/Write", "The number of times PGMPhysWrite was called.");
1735	PGM_REG_COUNTER_BYTES(&pStats->StatRCPhysWriteBytes, "/PGM/RC/Phys/Write/Bytes", "The number of bytes written by PGMPhysWrite.");
1736	PGM_REG_COUNTER(&pStats->StatRCPhysSimpleRead, "/PGM/RC/Phys/Simple/Read", "The number of times PGMPhysSimpleReadGCPtr was called.");
1737	PGM_REG_COUNTER_BYTES(&pStats->StatRCPhysSimpleReadBytes, "/PGM/RC/Phys/Simple/Read/Bytes", "The number of bytes read by PGMPhysSimpleReadGCPtr.");
1738	PGM_REG_COUNTER(&pStats->StatRCPhysSimpleWrite, "/PGM/RC/Phys/Simple/Write", "The number of times PGMPhysSimpleWriteGCPtr was called.");
1739	PGM_REG_COUNTER_BYTES(&pStats->StatRCPhysSimpleWriteBytes, "/PGM/RC/Phys/Simple/Write/Bytes", "The number of bytes written by PGMPhysSimpleWriteGCPtr.");
1740
1741	PGM_REG_COUNTER(&pStats->StatTrackVirgin, "/PGM/Track/Virgin", "The number of first time shadowings");
1742	PGM_REG_COUNTER(&pStats->StatTrackAliased, "/PGM/Track/Aliased", "The number of times switching to cRef2, i.e. the page is being shadowed by two PTs.");
1743	PGM_REG_COUNTER(&pStats->StatTrackAliasedMany, "/PGM/Track/AliasedMany", "The number of times we're tracking using cRef2.");
1744	PGM_REG_COUNTER(&pStats->StatTrackAliasedLots, "/PGM/Track/AliasedLots", "The number of times we're hitting pages which has overflowed cRef2");
1745	PGM_REG_COUNTER(&pStats->StatTrackOverflows, "/PGM/Track/Overflows", "The number of times the extent list grows too long.");
1746	PGM_REG_COUNTER(&pStats->StatTrackNoExtentsLeft, "/PGM/Track/NoExtentLeft", "The number of times the extent list was exhausted.");
1747	PGM_REG_PROFILE(&pStats->StatTrackDeref, "/PGM/Track/Deref", "Profiling of SyncPageWorkerTrackDeref (expensive).");
1748
1749	# undef PGM_REG_COUNTER
1750	# undef PGM_REG_PROFILE
1751	#endif
1752
1753	/*
1754	* Note! The layout below matches the member layout exactly!
1755	*/
1756
1757	/*
1758	* Common - stats
1759	*/
1760	for (VMCPUID idCpu = 0; idCpu < pVM->cCpus; idCpu++)
1761	{
1762	PPGMCPU pPgmCpu = &pVM->aCpus[idCpu].pgm.s;
1763
1764	#define PGM_REG_COUNTER(a, b, c) \
1765	rc = STAMR3RegisterF(pVM, a, STAMTYPE_COUNTER, STAMVISIBILITY_ALWAYS, STAMUNIT_OCCURENCES, c, b, idCpu); \
1766	AssertRC(rc);
1767	#define PGM_REG_PROFILE(a, b, c) \
1768	rc = STAMR3RegisterF(pVM, a, STAMTYPE_PROFILE, STAMVISIBILITY_ALWAYS, STAMUNIT_TICKS_PER_CALL, c, b, idCpu); \
1769	AssertRC(rc);
1770
1771	PGM_REG_COUNTER(&pPgmCpu->cGuestModeChanges, "/PGM/CPU%u/cGuestModeChanges", "Number of guest mode changes.");
1772
1773	#ifdef VBOX_WITH_STATISTICS
1774	PGMCPUSTATS *pCpuStats = pVM->aCpus[idCpu].pgm.s.pStatsR3;
1775
1776	# if 0 /* rarely useful; leave for debugging. */
1777	for (unsigned j = 0; j < RT_ELEMENTS(pPgmCpu->StatSyncPtPD); j++)
1778	STAMR3RegisterF(pVM, &pCpuStats->StatSyncPtPD[i], STAMTYPE_COUNTER, STAMVISIBILITY_USED, STAMUNIT_OCCURENCES,
1779	"The number of SyncPT per PD n.", "/PGM/CPU%u/PDSyncPT/%04X", i, j);
1780	for (unsigned j = 0; j < RT_ELEMENTS(pCpuStats->StatSyncPagePD); j++)
1781	STAMR3RegisterF(pVM, &pCpuStats->StatSyncPagePD[i], STAMTYPE_COUNTER, STAMVISIBILITY_USED, STAMUNIT_OCCURENCES,
1782	"The number of SyncPage per PD n.", "/PGM/CPU%u/PDSyncPage/%04X", i, j);
1783	# endif
1784	/* R0 only: */
1785	PGM_REG_PROFILE(&pCpuStats->StatR0NpMiscfg, "/PGM/CPU%u/R0/NpMiscfg", "PGMR0Trap0eHandlerNPMisconfig() profiling.");
1786	PGM_REG_COUNTER(&pCpuStats->StatR0NpMiscfgSyncPage, "/PGM/CPU%u/R0/NpMiscfgSyncPage", "SyncPage calls from PGMR0Trap0eHandlerNPMisconfig().");
1787
1788	/* RZ only: */
1789	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0e, "/PGM/CPU%u/RZ/Trap0e", "Profiling of the PGMTrap0eHandler() body.");
1790	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2Ballooned, "/PGM/CPU%u/RZ/Trap0e/Time2/Ballooned", "Profiling of the Trap0eHandler body when the cause is read access to a ballooned page.");
1791	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2CSAM, "/PGM/CPU%u/RZ/Trap0e/Time2/CSAM", "Profiling of the Trap0eHandler body when the cause is CSAM.");
1792	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2DirtyAndAccessed, "/PGM/CPU%u/RZ/Trap0e/Time2/DirtyAndAccessedBits", "Profiling of the Trap0eHandler body when the cause is dirty and/or accessed bit emulation.");
1793	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2GuestTrap, "/PGM/CPU%u/RZ/Trap0e/Time2/GuestTrap", "Profiling of the Trap0eHandler body when the cause is a guest trap.");
1794	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2HndPhys, "/PGM/CPU%u/RZ/Trap0e/Time2/HandlerPhysical", "Profiling of the Trap0eHandler body when the cause is a physical handler.");
1795	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2HndVirt, "/PGM/CPU%u/RZ/Trap0e/Time2/HandlerVirtual", "Profiling of the Trap0eHandler body when the cause is a virtual handler.");
1796	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2HndUnhandled, "/PGM/CPU%u/RZ/Trap0e/Time2/HandlerUnhandled", "Profiling of the Trap0eHandler body when the cause is access outside the monitored areas of a monitored page.");
1797	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2InvalidPhys, "/PGM/CPU%u/RZ/Trap0e/Time2/InvalidPhys", "Profiling of the Trap0eHandler body when the cause is access to an invalid physical guest address.");
1798	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2MakeWritable, "/PGM/CPU%u/RZ/Trap0e/Time2/MakeWritable", "Profiling of the Trap0eHandler body when the cause is that a page needed to be made writeable.");
1799	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2Mapping, "/PGM/CPU%u/RZ/Trap0e/Time2/Mapping", "Profiling of the Trap0eHandler body when the cause is related to the guest mappings.");
1800	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2Misc, "/PGM/CPU%u/RZ/Trap0e/Time2/Misc", "Profiling of the Trap0eHandler body when the cause is not known.");
1801	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2OutOfSync, "/PGM/CPU%u/RZ/Trap0e/Time2/OutOfSync", "Profiling of the Trap0eHandler body when the cause is an out-of-sync page.");
1802	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2OutOfSyncHndPhys, "/PGM/CPU%u/RZ/Trap0e/Time2/OutOfSyncHndPhys", "Profiling of the Trap0eHandler body when the cause is an out-of-sync physical handler page.");
1803	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2OutOfSyncHndVirt, "/PGM/CPU%u/RZ/Trap0e/Time2/OutOfSyncHndVirt", "Profiling of the Trap0eHandler body when the cause is an out-of-sync virtual handler page.");
1804	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2OutOfSyncHndObs, "/PGM/CPU%u/RZ/Trap0e/Time2/OutOfSyncObsHnd", "Profiling of the Trap0eHandler body when the cause is an obsolete handler page.");
1805	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2SyncPT, "/PGM/CPU%u/RZ/Trap0e/Time2/SyncPT", "Profiling of the Trap0eHandler body when the cause is lazy syncing of a PT.");
1806	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2WPEmulation, "/PGM/CPU%u/RZ/Trap0e/Time2/WPEmulation", "Profiling of the Trap0eHandler body when the cause is CR0.WP emulation.");
1807	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eConflicts, "/PGM/CPU%u/RZ/Trap0e/Conflicts", "The number of times #PF was caused by an undetected conflict.");
1808	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersMapping, "/PGM/CPU%u/RZ/Trap0e/Handlers/Mapping", "Number of traps due to access handlers in mappings.");
1809	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersOutOfSync, "/PGM/CPU%u/RZ/Trap0e/Handlers/OutOfSync", "Number of traps due to out-of-sync handled pages.");
1810	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersPhysAll, "/PGM/CPU%u/RZ/Trap0e/Handlers/PhysAll", "Number of traps due to physical all-access handlers.");
1811	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersPhysAllOpt, "/PGM/CPU%u/RZ/Trap0e/Handlers/PhysAllOpt", "Number of the physical all-access handler traps using the optimization.");
1812	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersPhysWrite, "/PGM/CPU%u/RZ/Trap0e/Handlers/PhysWrite", "Number of traps due to physical write-access handlers.");
1813	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersVirtual, "/PGM/CPU%u/RZ/Trap0e/Handlers/Virtual", "Number of traps due to virtual access handlers.");
1814	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersVirtualByPhys, "/PGM/CPU%u/RZ/Trap0e/Handlers/VirtualByPhys", "Number of traps due to virtual access handlers by physical address.");
1815	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersVirtualUnmarked,"/PGM/CPU%u/RZ/Trap0e/Handlers/VirtualUnmarked","Number of traps due to virtual access handlers by virtual address (without proper physical flags).");
1816	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersUnhandled, "/PGM/CPU%u/RZ/Trap0e/Handlers/Unhandled", "Number of traps due to access outside range of monitored page(s).");
1817	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersInvalid, "/PGM/CPU%u/RZ/Trap0e/Handlers/Invalid", "Number of traps due to access to invalid physical memory.");
1818	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eUSNotPresentRead, "/PGM/CPU%u/RZ/Trap0e/Err/User/NPRead", "Number of user mode not present read page faults.");
1819	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eUSNotPresentWrite, "/PGM/CPU%u/RZ/Trap0e/Err/User/NPWrite", "Number of user mode not present write page faults.");
1820	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eUSWrite, "/PGM/CPU%u/RZ/Trap0e/Err/User/Write", "Number of user mode write page faults.");
1821	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eUSReserved, "/PGM/CPU%u/RZ/Trap0e/Err/User/Reserved", "Number of user mode reserved bit page faults.");
1822	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eUSNXE, "/PGM/CPU%u/RZ/Trap0e/Err/User/NXE", "Number of user mode NXE page faults.");
1823	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eUSRead, "/PGM/CPU%u/RZ/Trap0e/Err/User/Read", "Number of user mode read page faults.");
1824	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eSVNotPresentRead, "/PGM/CPU%u/RZ/Trap0e/Err/Supervisor/NPRead", "Number of supervisor mode not present read page faults.");
1825	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eSVNotPresentWrite, "/PGM/CPU%u/RZ/Trap0e/Err/Supervisor/NPWrite", "Number of supervisor mode not present write page faults.");
1826	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eSVWrite, "/PGM/CPU%u/RZ/Trap0e/Err/Supervisor/Write", "Number of supervisor mode write page faults.");
1827	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eSVReserved, "/PGM/CPU%u/RZ/Trap0e/Err/Supervisor/Reserved", "Number of supervisor mode reserved bit page faults.");
1828	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eSNXE, "/PGM/CPU%u/RZ/Trap0e/Err/Supervisor/NXE", "Number of supervisor mode NXE page faults.");
1829	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eGuestPF, "/PGM/CPU%u/RZ/Trap0e/GuestPF", "Number of real guest page faults.");
1830	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eGuestPFMapping, "/PGM/CPU%u/RZ/Trap0e/GuestPF/InMapping", "Number of real guest page faults in a mapping.");
1831	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eWPEmulInRZ, "/PGM/CPU%u/RZ/Trap0e/WP/InRZ", "Number of guest page faults due to X86_CR0_WP emulation.");
1832	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eWPEmulToR3, "/PGM/CPU%u/RZ/Trap0e/WP/ToR3", "Number of guest page faults due to X86_CR0_WP emulation (forward to R3 for emulation).");
1833	#if 0 /* rarely useful; leave for debugging. */
1834	for (unsigned j = 0; j < RT_ELEMENTS(pCpuStats->StatRZTrap0ePD); j++)
1835	STAMR3RegisterF(pVM, &pCpuStats->StatRZTrap0ePD[i], STAMTYPE_COUNTER, STAMVISIBILITY_USED, STAMUNIT_OCCURENCES,
1836	"The number of traps in page directory n.", "/PGM/CPU%u/RZ/Trap0e/PD/%04X", i, j);
1837	#endif
1838	PGM_REG_COUNTER(&pCpuStats->StatRZGuestCR3WriteHandled, "/PGM/CPU%u/RZ/CR3WriteHandled", "The number of times the Guest CR3 change was successfully handled.");
1839	PGM_REG_COUNTER(&pCpuStats->StatRZGuestCR3WriteUnhandled, "/PGM/CPU%u/RZ/CR3WriteUnhandled", "The number of times the Guest CR3 change was passed back to the recompiler.");
1840	PGM_REG_COUNTER(&pCpuStats->StatRZGuestCR3WriteConflict, "/PGM/CPU%u/RZ/CR3WriteConflict", "The number of times the Guest CR3 monitoring detected a conflict.");
1841	PGM_REG_COUNTER(&pCpuStats->StatRZGuestROMWriteHandled, "/PGM/CPU%u/RZ/ROMWriteHandled", "The number of times the Guest ROM change was successfully handled.");
1842	PGM_REG_COUNTER(&pCpuStats->StatRZGuestROMWriteUnhandled, "/PGM/CPU%u/RZ/ROMWriteUnhandled", "The number of times the Guest ROM change was passed back to the recompiler.");
1843
1844	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapMigrateInvlPg, "/PGM/CPU%u/RZ/DynMap/MigrateInvlPg", "invlpg count in PGMR0DynMapMigrateAutoSet.");
1845	PGM_REG_PROFILE(&pCpuStats->StatRZDynMapGCPageInl, "/PGM/CPU%u/RZ/DynMap/PageGCPageInl", "Calls to pgmR0DynMapGCPageInlined.");
1846	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapGCPageInlHits, "/PGM/CPU%u/RZ/DynMap/PageGCPageInl/Hits", "Hash table lookup hits.");
1847	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapGCPageInlMisses, "/PGM/CPU%u/RZ/DynMap/PageGCPageInl/Misses", "Misses that falls back to the code common.");
1848	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapGCPageInlRamHits, "/PGM/CPU%u/RZ/DynMap/PageGCPageInl/RamHits", "1st ram range hits.");
1849	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapGCPageInlRamMisses, "/PGM/CPU%u/RZ/DynMap/PageGCPageInl/RamMisses", "1st ram range misses, takes slow path.");
1850	PGM_REG_PROFILE(&pCpuStats->StatRZDynMapHCPageInl, "/PGM/CPU%u/RZ/DynMap/PageHCPageInl", "Calls to pgmRZDynMapHCPageInlined.");
1851	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapHCPageInlHits, "/PGM/CPU%u/RZ/DynMap/PageHCPageInl/Hits", "Hash table lookup hits.");
1852	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapHCPageInlMisses, "/PGM/CPU%u/RZ/DynMap/PageHCPageInl/Misses", "Misses that falls back to the code common.");
1853	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPage, "/PGM/CPU%u/RZ/DynMap/Page", "Calls to pgmR0DynMapPage");
1854	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapSetOptimize, "/PGM/CPU%u/RZ/DynMap/Page/SetOptimize", "Calls to pgmRZDynMapOptimizeAutoSet.");
1855	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapSetSearchFlushes, "/PGM/CPU%u/RZ/DynMap/Page/SetSearchFlushes", "Set search restoring to subset flushes.");
1856	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapSetSearchHits, "/PGM/CPU%u/RZ/DynMap/Page/SetSearchHits", "Set search hits.");
1857	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapSetSearchMisses, "/PGM/CPU%u/RZ/DynMap/Page/SetSearchMisses", "Set search misses.");
1858	PGM_REG_PROFILE(&pCpuStats->StatRZDynMapHCPage, "/PGM/CPU%u/RZ/DynMap/Page/HCPage", "Calls to pgmRZDynMapHCPageCommon (ring-0).");
1859	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageHits0, "/PGM/CPU%u/RZ/DynMap/Page/Hits0", "Hits at iPage+0");
1860	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageHits1, "/PGM/CPU%u/RZ/DynMap/Page/Hits1", "Hits at iPage+1");
1861	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageHits2, "/PGM/CPU%u/RZ/DynMap/Page/Hits2", "Hits at iPage+2");
1862	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageInvlPg, "/PGM/CPU%u/RZ/DynMap/Page/InvlPg", "invlpg count in pgmR0DynMapPageSlow.");
1863	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageSlow, "/PGM/CPU%u/RZ/DynMap/Page/Slow", "Calls to pgmR0DynMapPageSlow - subtract this from pgmR0DynMapPage to get 1st level hits.");
1864	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageSlowLoopHits, "/PGM/CPU%u/RZ/DynMap/Page/SlowLoopHits" , "Hits in the loop path.");
1865	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageSlowLoopMisses, "/PGM/CPU%u/RZ/DynMap/Page/SlowLoopMisses", "Misses in the loop path. NonLoopMisses = Slow - SlowLoopHit - SlowLoopMisses");
1866	//PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageSlowLostHits, "/PGM/CPU%u/R0/DynMap/Page/SlowLostHits", "Lost hits.");
1867	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapSubsets, "/PGM/CPU%u/RZ/DynMap/Subsets", "Times PGMRZDynMapPushAutoSubset was called.");
1868	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPopFlushes, "/PGM/CPU%u/RZ/DynMap/SubsetPopFlushes", "Times PGMRZDynMapPopAutoSubset flushes the subset.");
1869	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[0], "/PGM/CPU%u/RZ/DynMap/SetFilledPct000..09", "00-09% filled (RC: min(set-size, dynmap-size))");
1870	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[1], "/PGM/CPU%u/RZ/DynMap/SetFilledPct010..19", "10-19% filled (RC: min(set-size, dynmap-size))");
1871	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[2], "/PGM/CPU%u/RZ/DynMap/SetFilledPct020..29", "20-29% filled (RC: min(set-size, dynmap-size))");
1872	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[3], "/PGM/CPU%u/RZ/DynMap/SetFilledPct030..39", "30-39% filled (RC: min(set-size, dynmap-size))");
1873	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[4], "/PGM/CPU%u/RZ/DynMap/SetFilledPct040..49", "40-49% filled (RC: min(set-size, dynmap-size))");
1874	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[5], "/PGM/CPU%u/RZ/DynMap/SetFilledPct050..59", "50-59% filled (RC: min(set-size, dynmap-size))");
1875	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[6], "/PGM/CPU%u/RZ/DynMap/SetFilledPct060..69", "60-69% filled (RC: min(set-size, dynmap-size))");
1876	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[7], "/PGM/CPU%u/RZ/DynMap/SetFilledPct070..79", "70-79% filled (RC: min(set-size, dynmap-size))");
1877	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[8], "/PGM/CPU%u/RZ/DynMap/SetFilledPct080..89", "80-89% filled (RC: min(set-size, dynmap-size))");
1878	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[9], "/PGM/CPU%u/RZ/DynMap/SetFilledPct090..99", "90-99% filled (RC: min(set-size, dynmap-size))");
1879	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[10], "/PGM/CPU%u/RZ/DynMap/SetFilledPct100", "100% filled (RC: min(set-size, dynmap-size))");
1880
1881	/* HC only: */
1882
1883	/* RZ & R3: */
1884	PGM_REG_PROFILE(&pCpuStats->StatRZSyncCR3, "/PGM/CPU%u/RZ/SyncCR3", "Profiling of the PGMSyncCR3() body.");
1885	PGM_REG_PROFILE(&pCpuStats->StatRZSyncCR3Handlers, "/PGM/CPU%u/RZ/SyncCR3/Handlers", "Profiling of the PGMSyncCR3() update handler section.");
1886	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3Global, "/PGM/CPU%u/RZ/SyncCR3/Global", "The number of global CR3 syncs.");
1887	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3NotGlobal, "/PGM/CPU%u/RZ/SyncCR3/NotGlobal", "The number of non-global CR3 syncs.");
1888	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3DstCacheHit, "/PGM/CPU%u/RZ/SyncCR3/DstChacheHit", "The number of times we got some kind of a cache hit.");
1889	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3DstFreed, "/PGM/CPU%u/RZ/SyncCR3/DstFreed", "The number of times we've had to free a shadow entry.");
1890	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3DstFreedSrcNP, "/PGM/CPU%u/RZ/SyncCR3/DstFreedSrcNP", "The number of times we've had to free a shadow entry for which the source entry was not present.");
1891	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3DstNotPresent, "/PGM/CPU%u/RZ/SyncCR3/DstNotPresent", "The number of times we've encountered a not present shadow entry for a present guest entry.");
1892	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3DstSkippedGlobalPD, "/PGM/CPU%u/RZ/SyncCR3/DstSkippedGlobalPD", "The number of times a global page directory wasn't flushed.");
1893	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3DstSkippedGlobalPT, "/PGM/CPU%u/RZ/SyncCR3/DstSkippedGlobalPT", "The number of times a page table with only global entries wasn't flushed.");
1894	PGM_REG_PROFILE(&pCpuStats->StatRZSyncPT, "/PGM/CPU%u/RZ/SyncPT", "Profiling of the pfnSyncPT() body.");
1895	PGM_REG_COUNTER(&pCpuStats->StatRZSyncPTFailed, "/PGM/CPU%u/RZ/SyncPT/Failed", "The number of times pfnSyncPT() failed.");
1896	PGM_REG_COUNTER(&pCpuStats->StatRZSyncPT4K, "/PGM/CPU%u/RZ/SyncPT/4K", "Nr of 4K PT syncs");
1897	PGM_REG_COUNTER(&pCpuStats->StatRZSyncPT4M, "/PGM/CPU%u/RZ/SyncPT/4M", "Nr of 4M PT syncs");
1898	PGM_REG_COUNTER(&pCpuStats->StatRZSyncPagePDNAs, "/PGM/CPU%u/RZ/SyncPagePDNAs", "The number of time we've marked a PD not present from SyncPage to virtualize the accessed bit.");
1899	PGM_REG_COUNTER(&pCpuStats->StatRZSyncPagePDOutOfSync, "/PGM/CPU%u/RZ/SyncPagePDOutOfSync", "The number of time we've encountered an out-of-sync PD in SyncPage.");
1900	PGM_REG_COUNTER(&pCpuStats->StatRZAccessedPage, "/PGM/CPU%u/RZ/AccessedPage", "The number of pages marked not present for accessed bit emulation.");
1901	PGM_REG_PROFILE(&pCpuStats->StatRZDirtyBitTracking, "/PGM/CPU%u/RZ/DirtyPage", "Profiling the dirty bit tracking in CheckPageFault().");
1902	PGM_REG_COUNTER(&pCpuStats->StatRZDirtyPage, "/PGM/CPU%u/RZ/DirtyPage/Mark", "The number of pages marked read-only for dirty bit tracking.");
1903	PGM_REG_COUNTER(&pCpuStats->StatRZDirtyPageBig, "/PGM/CPU%u/RZ/DirtyPage/MarkBig", "The number of 4MB pages marked read-only for dirty bit tracking.");
1904	PGM_REG_COUNTER(&pCpuStats->StatRZDirtyPageSkipped, "/PGM/CPU%u/RZ/DirtyPage/Skipped", "The number of pages already dirty or readonly.");
1905	PGM_REG_COUNTER(&pCpuStats->StatRZDirtyPageTrap, "/PGM/CPU%u/RZ/DirtyPage/Trap", "The number of traps generated for dirty bit tracking.");
1906	PGM_REG_COUNTER(&pCpuStats->StatRZDirtyPageStale, "/PGM/CPU%u/RZ/DirtyPage/Stale", "The number of traps generated for dirty bit tracking (stale tlb entries).");
1907	PGM_REG_COUNTER(&pCpuStats->StatRZDirtiedPage, "/PGM/CPU%u/RZ/DirtyPage/SetDirty", "The number of pages marked dirty because of write accesses.");
1908	PGM_REG_COUNTER(&pCpuStats->StatRZDirtyTrackRealPF, "/PGM/CPU%u/RZ/DirtyPage/RealPF", "The number of real pages faults during dirty bit tracking.");
1909	PGM_REG_COUNTER(&pCpuStats->StatRZPageAlreadyDirty, "/PGM/CPU%u/RZ/DirtyPage/AlreadySet", "The number of pages already marked dirty because of write accesses.");
1910	PGM_REG_PROFILE(&pCpuStats->StatRZInvalidatePage, "/PGM/CPU%u/RZ/InvalidatePage", "PGMInvalidatePage() profiling.");
1911	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePage4KBPages, "/PGM/CPU%u/RZ/InvalidatePage/4KBPages", "The number of times PGMInvalidatePage() was called for a 4KB page.");
1912	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePage4MBPages, "/PGM/CPU%u/RZ/InvalidatePage/4MBPages", "The number of times PGMInvalidatePage() was called for a 4MB page.");
1913	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePage4MBPagesSkip, "/PGM/CPU%u/RZ/InvalidatePage/4MBPagesSkip","The number of times PGMInvalidatePage() skipped a 4MB page.");
1914	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePagePDMappings, "/PGM/CPU%u/RZ/InvalidatePage/PDMappings", "The number of times PGMInvalidatePage() was called for a page directory containing mappings (no conflict).");
1915	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePagePDNAs, "/PGM/CPU%u/RZ/InvalidatePage/PDNAs", "The number of times PGMInvalidatePage() was called for a not accessed page directory.");
1916	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePagePDNPs, "/PGM/CPU%u/RZ/InvalidatePage/PDNPs", "The number of times PGMInvalidatePage() was called for a not present page directory.");
1917	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePagePDOutOfSync, "/PGM/CPU%u/RZ/InvalidatePage/PDOutOfSync", "The number of times PGMInvalidatePage() was called for an out of sync page directory.");
1918	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePageSkipped, "/PGM/CPU%u/RZ/InvalidatePage/Skipped", "The number of times PGMInvalidatePage() was skipped due to not present shw or pending pending SyncCR3.");
1919	PGM_REG_COUNTER(&pCpuStats->StatRZPageOutOfSyncSupervisor, "/PGM/CPU%u/RZ/OutOfSync/SuperVisor", "Number of traps due to pages out of sync (P) and times VerifyAccessSyncPage calls SyncPage.");
1920	PGM_REG_COUNTER(&pCpuStats->StatRZPageOutOfSyncUser, "/PGM/CPU%u/RZ/OutOfSync/User", "Number of traps due to pages out of sync (P) and times VerifyAccessSyncPage calls SyncPage.");
1921	PGM_REG_COUNTER(&pCpuStats->StatRZPageOutOfSyncSupervisorWrite,"/PGM/CPU%u/RZ/OutOfSync/SuperVisorWrite", "Number of traps due to pages out of sync (RW) and times VerifyAccessSyncPage calls SyncPage.");
1922	PGM_REG_COUNTER(&pCpuStats->StatRZPageOutOfSyncUserWrite, "/PGM/CPU%u/RZ/OutOfSync/UserWrite", "Number of traps due to pages out of sync (RW) and times VerifyAccessSyncPage calls SyncPage.");
1923	PGM_REG_COUNTER(&pCpuStats->StatRZPageOutOfSyncBallloon, "/PGM/CPU%u/RZ/OutOfSync/Balloon", "The number of times a ballooned page was accessed (read).");
1924	PGM_REG_PROFILE(&pCpuStats->StatRZPrefetch, "/PGM/CPU%u/RZ/Prefetch", "PGMPrefetchPage profiling.");
1925	PGM_REG_PROFILE(&pCpuStats->StatRZFlushTLB, "/PGM/CPU%u/RZ/FlushTLB", "Profiling of the PGMFlushTLB() body.");
1926	PGM_REG_COUNTER(&pCpuStats->StatRZFlushTLBNewCR3, "/PGM/CPU%u/RZ/FlushTLB/NewCR3", "The number of times PGMFlushTLB was called with a new CR3, non-global. (switch)");
1927	PGM_REG_COUNTER(&pCpuStats->StatRZFlushTLBNewCR3Global, "/PGM/CPU%u/RZ/FlushTLB/NewCR3Global", "The number of times PGMFlushTLB was called with a new CR3, global. (switch)");
1928	PGM_REG_COUNTER(&pCpuStats->StatRZFlushTLBSameCR3, "/PGM/CPU%u/RZ/FlushTLB/SameCR3", "The number of times PGMFlushTLB was called with the same CR3, non-global. (flush)");
1929	PGM_REG_COUNTER(&pCpuStats->StatRZFlushTLBSameCR3Global, "/PGM/CPU%u/RZ/FlushTLB/SameCR3Global", "The number of times PGMFlushTLB was called with the same CR3, global. (flush)");
1930	PGM_REG_PROFILE(&pCpuStats->StatRZGstModifyPage, "/PGM/CPU%u/RZ/GstModifyPage", "Profiling of the PGMGstModifyPage() body.");
1931
1932	PGM_REG_PROFILE(&pCpuStats->StatR3SyncCR3, "/PGM/CPU%u/R3/SyncCR3", "Profiling of the PGMSyncCR3() body.");
1933	PGM_REG_PROFILE(&pCpuStats->StatR3SyncCR3Handlers, "/PGM/CPU%u/R3/SyncCR3/Handlers", "Profiling of the PGMSyncCR3() update handler section.");
1934	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3Global, "/PGM/CPU%u/R3/SyncCR3/Global", "The number of global CR3 syncs.");
1935	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3NotGlobal, "/PGM/CPU%u/R3/SyncCR3/NotGlobal", "The number of non-global CR3 syncs.");
1936	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3DstCacheHit, "/PGM/CPU%u/R3/SyncCR3/DstChacheHit", "The number of times we got some kind of a cache hit.");
1937	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3DstFreed, "/PGM/CPU%u/R3/SyncCR3/DstFreed", "The number of times we've had to free a shadow entry.");
1938	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3DstFreedSrcNP, "/PGM/CPU%u/R3/SyncCR3/DstFreedSrcNP", "The number of times we've had to free a shadow entry for which the source entry was not present.");
1939	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3DstNotPresent, "/PGM/CPU%u/R3/SyncCR3/DstNotPresent", "The number of times we've encountered a not present shadow entry for a present guest entry.");
1940	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3DstSkippedGlobalPD, "/PGM/CPU%u/R3/SyncCR3/DstSkippedGlobalPD", "The number of times a global page directory wasn't flushed.");
1941	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3DstSkippedGlobalPT, "/PGM/CPU%u/R3/SyncCR3/DstSkippedGlobalPT", "The number of times a page table with only global entries wasn't flushed.");
1942	PGM_REG_PROFILE(&pCpuStats->StatR3SyncPT, "/PGM/CPU%u/R3/SyncPT", "Profiling of the pfnSyncPT() body.");
1943	PGM_REG_COUNTER(&pCpuStats->StatR3SyncPTFailed, "/PGM/CPU%u/R3/SyncPT/Failed", "The number of times pfnSyncPT() failed.");
1944	PGM_REG_COUNTER(&pCpuStats->StatR3SyncPT4K, "/PGM/CPU%u/R3/SyncPT/4K", "Nr of 4K PT syncs");
1945	PGM_REG_COUNTER(&pCpuStats->StatR3SyncPT4M, "/PGM/CPU%u/R3/SyncPT/4M", "Nr of 4M PT syncs");
1946	PGM_REG_COUNTER(&pCpuStats->StatR3SyncPagePDNAs, "/PGM/CPU%u/R3/SyncPagePDNAs", "The number of time we've marked a PD not present from SyncPage to virtualize the accessed bit.");
1947	PGM_REG_COUNTER(&pCpuStats->StatR3SyncPagePDOutOfSync, "/PGM/CPU%u/R3/SyncPagePDOutOfSync", "The number of time we've encountered an out-of-sync PD in SyncPage.");
1948	PGM_REG_COUNTER(&pCpuStats->StatR3AccessedPage, "/PGM/CPU%u/R3/AccessedPage", "The number of pages marked not present for accessed bit emulation.");
1949	PGM_REG_PROFILE(&pCpuStats->StatR3DirtyBitTracking, "/PGM/CPU%u/R3/DirtyPage", "Profiling the dirty bit tracking in CheckPageFault().");
1950	PGM_REG_COUNTER(&pCpuStats->StatR3DirtyPage, "/PGM/CPU%u/R3/DirtyPage/Mark", "The number of pages marked read-only for dirty bit tracking.");
1951	PGM_REG_COUNTER(&pCpuStats->StatR3DirtyPageBig, "/PGM/CPU%u/R3/DirtyPage/MarkBig", "The number of 4MB pages marked read-only for dirty bit tracking.");
1952	PGM_REG_COUNTER(&pCpuStats->StatR3DirtyPageSkipped, "/PGM/CPU%u/R3/DirtyPage/Skipped", "The number of pages already dirty or readonly.");
1953	PGM_REG_COUNTER(&pCpuStats->StatR3DirtyPageTrap, "/PGM/CPU%u/R3/DirtyPage/Trap", "The number of traps generated for dirty bit tracking.");
1954	PGM_REG_COUNTER(&pCpuStats->StatR3DirtiedPage, "/PGM/CPU%u/R3/DirtyPage/SetDirty", "The number of pages marked dirty because of write accesses.");
1955	PGM_REG_COUNTER(&pCpuStats->StatR3DirtyTrackRealPF, "/PGM/CPU%u/R3/DirtyPage/RealPF", "The number of real pages faults during dirty bit tracking.");
1956	PGM_REG_COUNTER(&pCpuStats->StatR3PageAlreadyDirty, "/PGM/CPU%u/R3/DirtyPage/AlreadySet", "The number of pages already marked dirty because of write accesses.");
1957	PGM_REG_PROFILE(&pCpuStats->StatR3InvalidatePage, "/PGM/CPU%u/R3/InvalidatePage", "PGMInvalidatePage() profiling.");
1958	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePage4KBPages, "/PGM/CPU%u/R3/InvalidatePage/4KBPages", "The number of times PGMInvalidatePage() was called for a 4KB page.");
1959	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePage4MBPages, "/PGM/CPU%u/R3/InvalidatePage/4MBPages", "The number of times PGMInvalidatePage() was called for a 4MB page.");
1960	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePage4MBPagesSkip, "/PGM/CPU%u/R3/InvalidatePage/4MBPagesSkip","The number of times PGMInvalidatePage() skipped a 4MB page.");
1961	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePagePDMappings, "/PGM/CPU%u/R3/InvalidatePage/PDMappings", "The number of times PGMInvalidatePage() was called for a page directory containing mappings (no conflict).");
1962	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePagePDNAs, "/PGM/CPU%u/R3/InvalidatePage/PDNAs", "The number of times PGMInvalidatePage() was called for a not accessed page directory.");
1963	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePagePDNPs, "/PGM/CPU%u/R3/InvalidatePage/PDNPs", "The number of times PGMInvalidatePage() was called for a not present page directory.");
1964	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePagePDOutOfSync, "/PGM/CPU%u/R3/InvalidatePage/PDOutOfSync", "The number of times PGMInvalidatePage() was called for an out of sync page directory.");
1965	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePageSkipped, "/PGM/CPU%u/R3/InvalidatePage/Skipped", "The number of times PGMInvalidatePage() was skipped due to not present shw or pending pending SyncCR3.");
1966	PGM_REG_COUNTER(&pCpuStats->StatR3PageOutOfSyncSupervisor, "/PGM/CPU%u/R3/OutOfSync/SuperVisor", "Number of traps due to pages out of sync and times VerifyAccessSyncPage calls SyncPage.");
1967	PGM_REG_COUNTER(&pCpuStats->StatR3PageOutOfSyncUser, "/PGM/CPU%u/R3/OutOfSync/User", "Number of traps due to pages out of sync and times VerifyAccessSyncPage calls SyncPage.");
1968	PGM_REG_COUNTER(&pCpuStats->StatR3PageOutOfSyncBallloon, "/PGM/CPU%u/R3/OutOfSync/Balloon", "The number of times a ballooned page was accessed (read).");
1969	PGM_REG_PROFILE(&pCpuStats->StatR3Prefetch, "/PGM/CPU%u/R3/Prefetch", "PGMPrefetchPage profiling.");
1970	PGM_REG_PROFILE(&pCpuStats->StatR3FlushTLB, "/PGM/CPU%u/R3/FlushTLB", "Profiling of the PGMFlushTLB() body.");
1971	PGM_REG_COUNTER(&pCpuStats->StatR3FlushTLBNewCR3, "/PGM/CPU%u/R3/FlushTLB/NewCR3", "The number of times PGMFlushTLB was called with a new CR3, non-global. (switch)");
1972	PGM_REG_COUNTER(&pCpuStats->StatR3FlushTLBNewCR3Global, "/PGM/CPU%u/R3/FlushTLB/NewCR3Global", "The number of times PGMFlushTLB was called with a new CR3, global. (switch)");
1973	PGM_REG_COUNTER(&pCpuStats->StatR3FlushTLBSameCR3, "/PGM/CPU%u/R3/FlushTLB/SameCR3", "The number of times PGMFlushTLB was called with the same CR3, non-global. (flush)");
1974	PGM_REG_COUNTER(&pCpuStats->StatR3FlushTLBSameCR3Global, "/PGM/CPU%u/R3/FlushTLB/SameCR3Global", "The number of times PGMFlushTLB was called with the same CR3, global. (flush)");
1975	PGM_REG_PROFILE(&pCpuStats->StatR3GstModifyPage, "/PGM/CPU%u/R3/GstModifyPage", "Profiling of the PGMGstModifyPage() body.");
1976	#endif /* VBOX_WITH_STATISTICS */
1977
1978	#undef PGM_REG_PROFILE
1979	#undef PGM_REG_COUNTER
1980
1981	}
1982
1983	return VINF_SUCCESS;
1984	}
1985
1986
1987	/**
1988	* Init the PGM bits that rely on VMMR0 and MM to be fully initialized.
1989	*
1990	* The dynamic mapping area will also be allocated and initialized at this
1991	* time. We could allocate it during PGMR3Init of course, but the mapping
1992	* wouldn't be allocated at that time preventing us from setting up the
1993	* page table entries with the dummy page.
1994	*
1995	* @returns VBox status code.
1996	* @param pVM VM handle.
1997	*/
1998	VMMR3DECL(int) PGMR3InitDynMap(PVM pVM)
1999	{
2000	RTGCPTR GCPtr;
2001	int rc;
2002
2003	/*
2004	* Reserve space for the dynamic mappings.
2005	*/
2006	rc = MMR3HyperReserve(pVM, MM_HYPER_DYNAMIC_SIZE, "Dynamic mapping", &GCPtr);
2007	if (RT_SUCCESS(rc))
2008	pVM->pgm.s.pbDynPageMapBaseGC = GCPtr;
2009
2010	if ( RT_SUCCESS(rc)
2011	&& (pVM->pgm.s.pbDynPageMapBaseGC >> X86_PD_PAE_SHIFT) != ((pVM->pgm.s.pbDynPageMapBaseGC + MM_HYPER_DYNAMIC_SIZE - 1) >> X86_PD_PAE_SHIFT))
2012	{
2013	rc = MMR3HyperReserve(pVM, MM_HYPER_DYNAMIC_SIZE, "Dynamic mapping not crossing", &GCPtr);
2014	if (RT_SUCCESS(rc))
2015	pVM->pgm.s.pbDynPageMapBaseGC = GCPtr;
2016	}
2017	if (RT_SUCCESS(rc))
2018	{
2019	AssertRelease((pVM->pgm.s.pbDynPageMapBaseGC >> X86_PD_PAE_SHIFT) == ((pVM->pgm.s.pbDynPageMapBaseGC + MM_HYPER_DYNAMIC_SIZE - 1) >> X86_PD_PAE_SHIFT));
2020	MMR3HyperReserve(pVM, PAGE_SIZE, "fence", NULL);
2021	}
2022	return rc;
2023	}
2024
2025
2026	/**
2027	* Ring-3 init finalizing.
2028	*
2029	* @returns VBox status code.
2030	* @param pVM The VM handle.
2031	*/
2032	VMMR3DECL(int) PGMR3InitFinalize(PVM pVM)
2033	{
2034	int rc;
2035
2036	/*
2037	* Reserve space for the dynamic mappings.
2038	* Initialize the dynamic mapping pages with dummy pages to simply the cache.
2039	*/
2040	/* get the pointer to the page table entries. */
2041	PPGMMAPPING pMapping = pgmGetMapping(pVM, pVM->pgm.s.pbDynPageMapBaseGC);
2042	AssertRelease(pMapping);
2043	const uintptr_t off = pVM->pgm.s.pbDynPageMapBaseGC - pMapping->GCPtr;
2044	const unsigned iPT = off >> X86_PD_SHIFT;
2045	const unsigned iPG = (off >> X86_PT_SHIFT) & X86_PT_MASK;
2046	pVM->pgm.s.paDynPageMap32BitPTEsGC = pMapping->aPTs[iPT].pPTRC + iPG * sizeof(pMapping->aPTs[0].pPTR3->a[0]);
2047	pVM->pgm.s.paDynPageMapPaePTEsGC = pMapping->aPTs[iPT].paPaePTsRC + iPG * sizeof(pMapping->aPTs[0].paPaePTsR3->a[0]);
2048
2049	/* init cache area */
2050	RTHCPHYS HCPhysDummy = MMR3PageDummyHCPhys(pVM);
2051	for (uint32_t offDynMap = 0; offDynMap < MM_HYPER_DYNAMIC_SIZE; offDynMap += PAGE_SIZE)
2052	{
2053	rc = PGMMap(pVM, pVM->pgm.s.pbDynPageMapBaseGC + offDynMap, HCPhysDummy, PAGE_SIZE, 0);
2054	AssertRCReturn(rc, rc);
2055	}
2056
2057	/*
2058	* Determine the max physical address width (MAXPHYADDR) and apply it to
2059	* all the mask members and stuff.
2060	*/
2061	uint32_t cMaxPhysAddrWidth;
2062	uint32_t uMaxExtLeaf = ASMCpuId_EAX(0x80000000);
2063	if ( uMaxExtLeaf >= 0x80000008
2064	&& uMaxExtLeaf <= 0x80000fff)
2065	{
2066	cMaxPhysAddrWidth = ASMCpuId_EAX(0x80000008) & 0xff;
2067	LogRel(("PGM: The CPU physical address width is %u bits\n", cMaxPhysAddrWidth));
2068	cMaxPhysAddrWidth = RT_MIN(52, cMaxPhysAddrWidth);
2069	pVM->pgm.s.fLessThan52PhysicalAddressBits = cMaxPhysAddrWidth < 52;
2070	for (uint32_t iBit = cMaxPhysAddrWidth; iBit < 52; iBit++)
2071	pVM->pgm.s.HCPhysInvMmioPg \|= RT_BIT_64(iBit);
2072	}
2073	else
2074	{
2075	LogRel(("PGM: ASSUMING CPU physical address width of 48 bits (uMaxExtLeaf=%#x)\n", uMaxExtLeaf));
2076	cMaxPhysAddrWidth = 48;
2077	pVM->pgm.s.fLessThan52PhysicalAddressBits = true;
2078	pVM->pgm.s.HCPhysInvMmioPg \|= UINT64_C(0x000f0000000000);
2079	}
2080
2081	pVM->pgm.s.GCPhysInvAddrMask = 0;
2082	for (uint32_t iBit = cMaxPhysAddrWidth; iBit < 64; iBit++)
2083	pVM->pgm.s.GCPhysInvAddrMask \|= RT_BIT_64(iBit);
2084
2085	/*
2086	* Initialize the invalid paging entry masks, assuming NX is disabled.
2087	*/
2088	uint64_t fMbzPageFrameMask = pVM->pgm.s.GCPhysInvAddrMask & UINT64_C(0x000ffffffffff000);
2089	for (VMCPUID iCpu = 0; iCpu < pVM->cCpus; iCpu++)
2090	{
2091	PVMCPU pVCpu = &pVM->aCpus[iCpu];
2092
2093	/** @todo The manuals are not entirely clear whether the physical
2094	* address width is relevant. See table 5-9 in the intel
2095	* manual vs the PDE4M descriptions. Write testcase (NP). */
2096	pVCpu->pgm.s.fGst32BitMbzBigPdeMask = ((uint32_t)(fMbzPageFrameMask >> (32 - 13)) & X86_PDE4M_PG_HIGH_MASK)
2097	\| X86_PDE4M_MBZ_MASK;
2098
2099	pVCpu->pgm.s.fGstPaeMbzPteMask = fMbzPageFrameMask \| X86_PTE_PAE_MBZ_MASK_NO_NX;
2100	pVCpu->pgm.s.fGstPaeMbzPdeMask = fMbzPageFrameMask \| X86_PDE_PAE_MBZ_MASK_NO_NX;
2101	pVCpu->pgm.s.fGstPaeMbzBigPdeMask = fMbzPageFrameMask \| X86_PDE2M_PAE_MBZ_MASK_NO_NX;
2102	pVCpu->pgm.s.fGstPaeMbzPdpeMask = fMbzPageFrameMask \| X86_PDPE_PAE_MBZ_MASK;
2103
2104	pVCpu->pgm.s.fGstAmd64MbzPteMask = fMbzPageFrameMask \| X86_PTE_LM_MBZ_MASK_NO_NX;
2105	pVCpu->pgm.s.fGstAmd64MbzPdeMask = fMbzPageFrameMask \| X86_PDE_LM_MBZ_MASK_NX;
2106	pVCpu->pgm.s.fGstAmd64MbzBigPdeMask = fMbzPageFrameMask \| X86_PDE2M_LM_MBZ_MASK_NX;
2107	pVCpu->pgm.s.fGstAmd64MbzPdpeMask = fMbzPageFrameMask \| X86_PDPE_LM_MBZ_MASK_NO_NX;
2108	pVCpu->pgm.s.fGstAmd64MbzBigPdpeMask = fMbzPageFrameMask \| X86_PDPE1G_LM_MBZ_MASK_NO_NX;
2109	pVCpu->pgm.s.fGstAmd64MbzPml4eMask = fMbzPageFrameMask \| X86_PML4E_MBZ_MASK_NO_NX;
2110
2111	pVCpu->pgm.s.fGst64ShadowedPteMask = X86_PTE_P \| X86_PTE_RW \| X86_PTE_US \| X86_PTE_G \| X86_PTE_A \| X86_PTE_D;
2112	pVCpu->pgm.s.fGst64ShadowedPdeMask = X86_PDE_P \| X86_PDE_RW \| X86_PDE_US \| X86_PDE_A;
2113	pVCpu->pgm.s.fGst64ShadowedBigPdeMask = X86_PDE4M_P \| X86_PDE4M_RW \| X86_PDE4M_US \| X86_PDE4M_A;
2114	pVCpu->pgm.s.fGst64ShadowedBigPde4PteMask =
2115	X86_PDE4M_P \| X86_PDE4M_RW \| X86_PDE4M_US \| X86_PDE4M_G \| X86_PDE4M_A \| X86_PDE4M_D;
2116	pVCpu->pgm.s.fGstAmd64ShadowedPdpeMask = X86_PDPE_P \| X86_PDPE_RW \| X86_PDPE_US \| X86_PDPE_A;
2117	pVCpu->pgm.s.fGstAmd64ShadowedPml4eMask = X86_PML4E_P \| X86_PML4E_RW \| X86_PML4E_US \| X86_PML4E_A;
2118	}
2119
2120	/*
2121	* Note that AMD uses all the 8 reserved bits for the address (so 40 bits in total);
2122	* Intel only goes up to 36 bits, so we stick to 36 as well.
2123	* Update: More recent intel manuals specifies 40 bits just like AMD.
2124	*/
2125	uint32_t u32Dummy, u32Features;
2126	CPUMGetGuestCpuId(VMMGetCpu(pVM), 1, &u32Dummy, &u32Dummy, &u32Dummy, &u32Features);
2127	if (u32Features & X86_CPUID_FEATURE_EDX_PSE36)
2128	pVM->pgm.s.GCPhys4MBPSEMask = RT_BIT_64(RT_MAX(36, cMaxPhysAddrWidth)) - 1;
2129	else
2130	pVM->pgm.s.GCPhys4MBPSEMask = RT_BIT_64(32) - 1;
2131
2132	/*
2133	* Allocate memory if we're supposed to do that.
2134	*/
2135	if (pVM->pgm.s.fRamPreAlloc)
2136	rc = pgmR3PhysRamPreAllocate(pVM);
2137
2138	LogRel(("PGMR3InitFinalize: 4 MB PSE mask %RGp\n", pVM->pgm.s.GCPhys4MBPSEMask));
2139	return rc;
2140	}
2141
2142
2143	/**
2144	* Init phase completed callback.
2145	*
2146	* @returns VBox status code.
2147	* @param pVM The VM handle.
2148	* @param enmWhat What has been completed.
2149	* @thread EMT(0)
2150	*/
2151	VMMR3_INT_DECL(int) PGMR3InitCompleted(PVM pVM, VMINITCOMPLETED enmWhat)
2152	{
2153	switch (enmWhat)
2154	{
2155	case VMINITCOMPLETED_HWACCM:
2156	#ifdef VBOX_WITH_PCI_PASSTHROUGH
2157	if (pVM->pgm.s.fPciPassthrough)
2158	{
2159	AssertLogRelReturn(pVM->pgm.s.fRamPreAlloc, VERR_INVALID_PARAMETER);
2160	AssertLogRelReturn(HWACCMIsEnabled(pVM), VERR_INVALID_PARAMETER);
2161	AssertLogRelReturn(HWACCMIsNestedPagingActive(pVM), VERR_INVALID_PARAMETER);
2162
2163	/*
2164	* Report assignments to the IOMMU (hope that's good enough for now).
2165	*/
2166	if (pVM->pgm.s.fPciPassthrough)
2167	{
2168	int rc = VMMR3CallR0(pVM, VMMR0_DO_PGM_PHYS_SETUP_IOMMU, 0, NULL);
2169	AssertRCReturn(rc, rc);
2170	}
2171	}
2172	#else
2173	AssertLogRelReturn(!pVM->pgm.s.fPciPassthrough, VERR_INTERNAL_ERROR_5);
2174	#endif
2175	break;
2176
2177	default:
2178	/* shut up gcc */
2179	break;
2180	}
2181
2182	return VINF_SUCCESS;
2183	}
2184
2185
2186	/**
2187	* Applies relocations to data and code managed by this component.
2188	*
2189	* This function will be called at init and whenever the VMM need to relocate it
2190	* self inside the GC.
2191	*
2192	* @param pVM The VM.
2193	* @param offDelta Relocation delta relative to old location.
2194	*/
2195	VMMR3DECL(void) PGMR3Relocate(PVM pVM, RTGCINTPTR offDelta)
2196	{
2197	LogFlow(("PGMR3Relocate %RGv to %RGv\n", pVM->pgm.s.GCPtrCR3Mapping, pVM->pgm.s.GCPtrCR3Mapping + offDelta));
2198
2199	/*
2200	* Paging stuff.
2201	*/
2202	pVM->pgm.s.GCPtrCR3Mapping += offDelta;
2203
2204	pgmR3ModeDataInit(pVM, true /* resolve GC/R0 symbols */);
2205
2206	/* Shadow, guest and both mode switch & relocation for each VCPU. */
2207	for (VMCPUID i = 0; i < pVM->cCpus; i++)
2208	{
2209	PVMCPU pVCpu = &pVM->aCpus[i];
2210
2211	pgmR3ModeDataSwitch(pVM, pVCpu, pVCpu->pgm.s.enmShadowMode, pVCpu->pgm.s.enmGuestMode);
2212
2213	PGM_SHW_PFN(Relocate, pVCpu)(pVCpu, offDelta);
2214	PGM_GST_PFN(Relocate, pVCpu)(pVCpu, offDelta);
2215	PGM_BTH_PFN(Relocate, pVCpu)(pVCpu, offDelta);
2216	}
2217
2218	/*
2219	* Trees.
2220	*/
2221	pVM->pgm.s.pTreesRC = MMHyperR3ToRC(pVM, pVM->pgm.s.pTreesR3);
2222
2223	/*
2224	* Ram ranges.
2225	*/
2226	if (pVM->pgm.s.pRamRangesR3)
2227	{
2228	/* Update the pSelfRC pointers and relink them. */
2229	for (PPGMRAMRANGE pCur = pVM->pgm.s.pRamRangesR3; pCur; pCur = pCur->pNextR3)
2230	if (!(pCur->fFlags & PGM_RAM_RANGE_FLAGS_FLOATING))
2231	pCur->pSelfRC = MMHyperCCToRC(pVM, pCur);
2232	pgmR3PhysRelinkRamRanges(pVM);
2233	}
2234
2235	/*
2236	* Update the pSelfRC pointer of the MMIO2 ram ranges since they might not
2237	* be mapped and thus not included in the above exercise.
2238	*/
2239	for (PPGMMMIO2RANGE pCur = pVM->pgm.s.pMmio2RangesR3; pCur; pCur = pCur->pNextR3)
2240	if (!(pCur->RamRange.fFlags & PGM_RAM_RANGE_FLAGS_FLOATING))
2241	pCur->RamRange.pSelfRC = MMHyperCCToRC(pVM, &pCur->RamRange);
2242
2243	/*
2244	* Update the two page directories with all page table mappings.
2245	* (One or more of them have changed, that's why we're here.)
2246	*/
2247	pVM->pgm.s.pMappingsRC = MMHyperR3ToRC(pVM, pVM->pgm.s.pMappingsR3);
2248	for (PPGMMAPPING pCur = pVM->pgm.s.pMappingsR3; pCur->pNextR3; pCur = pCur->pNextR3)
2249	pCur->pNextRC = MMHyperR3ToRC(pVM, pCur->pNextR3);
2250
2251	/* Relocate GC addresses of Page Tables. */
2252	for (PPGMMAPPING pCur = pVM->pgm.s.pMappingsR3; pCur; pCur = pCur->pNextR3)
2253	{
2254	for (RTHCUINT i = 0; i < pCur->cPTs; i++)
2255	{
2256	pCur->aPTs[i].pPTRC = MMHyperR3ToRC(pVM, pCur->aPTs[i].pPTR3);
2257	pCur->aPTs[i].paPaePTsRC = MMHyperR3ToRC(pVM, pCur->aPTs[i].paPaePTsR3);
2258	}
2259	}
2260
2261	/*
2262	* Dynamic page mapping area.
2263	*/
2264	pVM->pgm.s.paDynPageMap32BitPTEsGC += offDelta;
2265	pVM->pgm.s.paDynPageMapPaePTEsGC += offDelta;
2266	pVM->pgm.s.pbDynPageMapBaseGC += offDelta;
2267
2268	if (pVM->pgm.s.pRCDynMap)
2269	{
2270	pVM->pgm.s.pRCDynMap += offDelta;
2271	PPGMRCDYNMAP pDynMap = (PPGMRCDYNMAP)MMHyperRCToCC(pVM, pVM->pgm.s.pRCDynMap);
2272
2273	pDynMap->paPages += offDelta;
2274	PPGMRCDYNMAPENTRY paPages = (PPGMRCDYNMAPENTRY)MMHyperRCToCC(pVM, pDynMap->paPages);
2275
2276	for (uint32_t iPage = 0; iPage < pDynMap->cPages; iPage++)
2277	{
2278	paPages[iPage].pvPage += offDelta;
2279	paPages[iPage].uPte.pLegacy += offDelta;
2280	paPages[iPage].uPte.pPae += offDelta;
2281	}
2282	}
2283
2284	/*
2285	* The Zero page.
2286	*/
2287	pVM->pgm.s.pvZeroPgR0 = MMHyperR3ToR0(pVM, pVM->pgm.s.pvZeroPgR3);
2288	#ifdef VBOX_WITH_2X_4GB_ADDR_SPACE
2289	AssertRelease(pVM->pgm.s.pvZeroPgR0 != NIL_RTR0PTR \|\| !VMMIsHwVirtExtForced(pVM));
2290	#else
2291	AssertRelease(pVM->pgm.s.pvZeroPgR0 != NIL_RTR0PTR);
2292	#endif
2293
2294	/*
2295	* Physical and virtual handlers.
2296	*/
2297	RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesR3->PhysHandlers, true, pgmR3RelocatePhysHandler, &offDelta);
2298	pVM->pgm.s.pLastPhysHandlerRC = NIL_RTRCPTR;
2299	RTAvlroGCPtrDoWithAll(&pVM->pgm.s.pTreesR3->VirtHandlers, true, pgmR3RelocateVirtHandler, &offDelta);
2300	RTAvlroGCPtrDoWithAll(&pVM->pgm.s.pTreesR3->HyperVirtHandlers, true, pgmR3RelocateHyperVirtHandler, &offDelta);
2301
2302	/*
2303	* The page pool.
2304	*/
2305	pgmR3PoolRelocate(pVM);
2306
2307	#ifdef VBOX_WITH_STATISTICS
2308	/*
2309	* Statistics.
2310	*/
2311	pVM->pgm.s.pStatsRC = MMHyperCCToRC(pVM, pVM->pgm.s.pStatsR3);
2312	for (VMCPUID iCpu = 0; iCpu < pVM->cCpus; iCpu++)
2313	pVM->aCpus[iCpu].pgm.s.pStatsRC = MMHyperCCToRC(pVM, pVM->aCpus[iCpu].pgm.s.pStatsR3);
2314	#endif
2315	}
2316
2317
2318	/**
2319	* Callback function for relocating a physical access handler.
2320	*
2321	* @returns 0 (continue enum)
2322	* @param pNode Pointer to a PGMPHYSHANDLER node.
2323	* @param pvUser Pointer to the offDelta. This is a pointer to the delta since we're
2324	* not certain the delta will fit in a void pointer for all possible configs.
2325	*/
2326	static DECLCALLBACK(int) pgmR3RelocatePhysHandler(PAVLROGCPHYSNODECORE pNode, void *pvUser)
2327	{
2328	PPGMPHYSHANDLER pHandler = (PPGMPHYSHANDLER)pNode;
2329	RTGCINTPTR offDelta = *(PRTGCINTPTR)pvUser;
2330	if (pHandler->pfnHandlerRC)
2331	pHandler->pfnHandlerRC += offDelta;
2332	if (pHandler->pvUserRC >= 0x10000)
2333	pHandler->pvUserRC += offDelta;
2334	return 0;
2335	}
2336
2337
2338	/**
2339	* Callback function for relocating a virtual access handler.
2340	*
2341	* @returns 0 (continue enum)
2342	* @param pNode Pointer to a PGMVIRTHANDLER node.
2343	* @param pvUser Pointer to the offDelta. This is a pointer to the delta since we're
2344	* not certain the delta will fit in a void pointer for all possible configs.
2345	*/
2346	static DECLCALLBACK(int) pgmR3RelocateVirtHandler(PAVLROGCPTRNODECORE pNode, void *pvUser)
2347	{
2348	PPGMVIRTHANDLER pHandler = (PPGMVIRTHANDLER)pNode;
2349	RTGCINTPTR offDelta = *(PRTGCINTPTR)pvUser;
2350	Assert( pHandler->enmType == PGMVIRTHANDLERTYPE_ALL
2351	\|\| pHandler->enmType == PGMVIRTHANDLERTYPE_WRITE);
2352	Assert(pHandler->pfnHandlerRC);
2353	pHandler->pfnHandlerRC += offDelta;
2354	return 0;
2355	}
2356
2357
2358	/**
2359	* Callback function for relocating a virtual access handler for the hypervisor mapping.
2360	*
2361	* @returns 0 (continue enum)
2362	* @param pNode Pointer to a PGMVIRTHANDLER node.
2363	* @param pvUser Pointer to the offDelta. This is a pointer to the delta since we're
2364	* not certain the delta will fit in a void pointer for all possible configs.
2365	*/
2366	static DECLCALLBACK(int) pgmR3RelocateHyperVirtHandler(PAVLROGCPTRNODECORE pNode, void *pvUser)
2367	{
2368	PPGMVIRTHANDLER pHandler = (PPGMVIRTHANDLER)pNode;
2369	RTGCINTPTR offDelta = *(PRTGCINTPTR)pvUser;
2370	Assert(pHandler->enmType == PGMVIRTHANDLERTYPE_HYPERVISOR);
2371	Assert(pHandler->pfnHandlerRC);
2372	pHandler->pfnHandlerRC += offDelta;
2373	return 0;
2374	}
2375
2376
2377	/**
2378	* Resets a virtual CPU when unplugged.
2379	*
2380	* @param pVM The VM handle.
2381	* @param pVCpu The virtual CPU handle.
2382	*/
2383	VMMR3DECL(void) PGMR3ResetUnpluggedCpu(PVM pVM, PVMCPU pVCpu)
2384	{
2385	int rc = PGM_GST_PFN(Exit, pVCpu)(pVCpu);
2386	AssertRC(rc);
2387
2388	rc = PGMR3ChangeMode(pVM, pVCpu, PGMMODE_REAL);
2389	AssertRC(rc);
2390
2391	STAM_REL_COUNTER_RESET(&pVCpu->pgm.s.cGuestModeChanges);
2392
2393	pgmR3PoolResetUnpluggedCpu(pVM, pVCpu);
2394
2395	/*
2396	* Re-init other members.
2397	*/
2398	pVCpu->pgm.s.fA20Enabled = true;
2399
2400	/*
2401	* Clear the FFs PGM owns.
2402	*/
2403	VMCPU_FF_CLEAR(pVCpu, VMCPU_FF_PGM_SYNC_CR3);
2404	VMCPU_FF_CLEAR(pVCpu, VMCPU_FF_PGM_SYNC_CR3_NON_GLOBAL);
2405	}
2406
2407
2408	/**
2409	* The VM is being reset.
2410	*
2411	* For the PGM component this means that any PD write monitors
2412	* needs to be removed.
2413	*
2414	* @param pVM VM handle.
2415	*/
2416	VMMR3DECL(void) PGMR3Reset(PVM pVM)
2417	{
2418	int rc;
2419
2420	LogFlow(("PGMR3Reset:\n"));
2421	VM_ASSERT_EMT(pVM);
2422
2423	pgmLock(pVM);
2424
2425	/*
2426	* Unfix any fixed mappings and disable CR3 monitoring.
2427	*/
2428	pVM->pgm.s.fMappingsFixed = false;
2429	pVM->pgm.s.fMappingsFixedRestored = false;
2430	pVM->pgm.s.GCPtrMappingFixed = NIL_RTGCPTR;
2431	pVM->pgm.s.cbMappingFixed = 0;
2432
2433	/*
2434	* Exit the guest paging mode before the pgm pool gets reset.
2435	* Important to clean up the amd64 case.
2436	*/
2437	for (VMCPUID i = 0; i < pVM->cCpus; i++)
2438	{
2439	PVMCPU pVCpu = &pVM->aCpus[i];
2440	rc = PGM_GST_PFN(Exit, pVCpu)(pVCpu);
2441	AssertRC(rc);
2442	}
2443
2444	#ifdef DEBUG
2445	DBGFR3InfoLog(pVM, "mappings", NULL);
2446	DBGFR3InfoLog(pVM, "handlers", "all nostat");
2447	#endif
2448
2449	/*
2450	* Switch mode back to real mode. (before resetting the pgm pool!)
2451	*/
2452	for (VMCPUID i = 0; i < pVM->cCpus; i++)
2453	{
2454	PVMCPU pVCpu = &pVM->aCpus[i];
2455
2456	rc = PGMR3ChangeMode(pVM, pVCpu, PGMMODE_REAL);
2457	AssertRC(rc);
2458
2459	STAM_REL_COUNTER_RESET(&pVCpu->pgm.s.cGuestModeChanges);
2460	}
2461
2462	/*
2463	* Reset the shadow page pool.
2464	*/
2465	pgmR3PoolReset(pVM);
2466
2467	/*
2468	* Re-init various other members and clear the FFs that PGM owns.
2469	*/
2470	for (VMCPUID i = 0; i < pVM->cCpus; i++)
2471	{
2472	PVMCPU pVCpu = &pVM->aCpus[i];
2473
2474	pVCpu->pgm.s.fA20Enabled = true;
2475	pVCpu->pgm.s.fGst32BitPageSizeExtension = false;
2476	PGMNotifyNxeChanged(pVCpu, false);
2477
2478	VMCPU_FF_CLEAR(pVCpu, VMCPU_FF_PGM_SYNC_CR3);
2479	VMCPU_FF_CLEAR(pVCpu, VMCPU_FF_PGM_SYNC_CR3_NON_GLOBAL);
2480	}
2481
2482	/*
2483	* Reset (zero) RAM and shadow ROM pages.
2484	*/
2485	rc = pgmR3PhysRamReset(pVM);
2486	if (RT_SUCCESS(rc))
2487	rc = pgmR3PhysRomReset(pVM);
2488
2489
2490	pgmUnlock(pVM);
2491	AssertReleaseRC(rc);
2492	}
2493
2494
2495	#ifdef VBOX_STRICT
2496	/**
2497	* VM state change callback for clearing fNoMorePhysWrites after
2498	* a snapshot has been created.
2499	*/
2500	static DECLCALLBACK(void) pgmR3ResetNoMorePhysWritesFlag(PVM pVM, VMSTATE enmState, VMSTATE enmOldState, void *pvUser)
2501	{
2502	if ( enmState == VMSTATE_RUNNING
2503	\|\| enmState == VMSTATE_RESUMING)
2504	pVM->pgm.s.fNoMorePhysWrites = false;
2505	}
2506	#endif
2507
2508	/**
2509	* Private API to reset fNoMorePhysWrites.
2510	*/
2511	VMMR3DECL(void) PGMR3ResetNoMorePhysWritesFlag(PVM pVM)
2512	{
2513	pVM->pgm.s.fNoMorePhysWrites = false;
2514	}
2515
2516	/**
2517	* Terminates the PGM.
2518	*
2519	* @returns VBox status code.
2520	* @param pVM Pointer to VM structure.
2521	*/
2522	VMMR3DECL(int) PGMR3Term(PVM pVM)
2523	{
2524	/* Must free shared pages here. */
2525	pgmLock(pVM);
2526	pgmR3PhysRamTerm(pVM);
2527	pgmR3PhysRomTerm(pVM);
2528	pgmUnlock(pVM);
2529
2530	PGMDeregisterStringFormatTypes();
2531	return PDMR3CritSectDelete(&pVM->pgm.s.CritSect);
2532	}
2533
2534
2535	/**
2536	* Show paging mode.
2537	*
2538	* @param pVM VM Handle.
2539	* @param pHlp The info helpers.
2540	* @param pszArgs "all" (default), "guest", "shadow" or "host".
2541	*/
2542	static DECLCALLBACK(void) pgmR3InfoMode(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs)
2543	{
2544	/* digest argument. */
2545	bool fGuest, fShadow, fHost;
2546	if (pszArgs)
2547	pszArgs = RTStrStripL(pszArgs);
2548	if (!pszArgs \|\| !*pszArgs \|\| strstr(pszArgs, "all"))
2549	fShadow = fHost = fGuest = true;
2550	else
2551	{
2552	fShadow = fHost = fGuest = false;
2553	if (strstr(pszArgs, "guest"))
2554	fGuest = true;
2555	if (strstr(pszArgs, "shadow"))
2556	fShadow = true;
2557	if (strstr(pszArgs, "host"))
2558	fHost = true;
2559	}
2560
2561	/** @todo SMP support! */
2562	/* print info. */
2563	if (fGuest)
2564	pHlp->pfnPrintf(pHlp, "Guest paging mode: %s, changed %RU64 times, A20 %s\n",
2565	PGMGetModeName(pVM->aCpus[0].pgm.s.enmGuestMode), pVM->aCpus[0].pgm.s.cGuestModeChanges.c,
2566	pVM->aCpus[0].pgm.s.fA20Enabled ? "enabled" : "disabled");
2567	if (fShadow)
2568	pHlp->pfnPrintf(pHlp, "Shadow paging mode: %s\n", PGMGetModeName(pVM->aCpus[0].pgm.s.enmShadowMode));
2569	if (fHost)
2570	{
2571	const char *psz;
2572	switch (pVM->pgm.s.enmHostMode)
2573	{
2574	case SUPPAGINGMODE_INVALID: psz = "invalid"; break;
2575	case SUPPAGINGMODE_32_BIT: psz = "32-bit"; break;
2576	case SUPPAGINGMODE_32_BIT_GLOBAL: psz = "32-bit+G"; break;
2577	case SUPPAGINGMODE_PAE: psz = "PAE"; break;
2578	case SUPPAGINGMODE_PAE_GLOBAL: psz = "PAE+G"; break;
2579	case SUPPAGINGMODE_PAE_NX: psz = "PAE+NX"; break;
2580	case SUPPAGINGMODE_PAE_GLOBAL_NX: psz = "PAE+G+NX"; break;
2581	case SUPPAGINGMODE_AMD64: psz = "AMD64"; break;
2582	case SUPPAGINGMODE_AMD64_GLOBAL: psz = "AMD64+G"; break;
2583	case SUPPAGINGMODE_AMD64_NX: psz = "AMD64+NX"; break;
2584	case SUPPAGINGMODE_AMD64_GLOBAL_NX: psz = "AMD64+G+NX"; break;
2585	default: psz = "unknown"; break;
2586	}
2587	pHlp->pfnPrintf(pHlp, "Host paging mode: %s\n", psz);
2588	}
2589	}
2590
2591
2592	/**
2593	* Dump registered MMIO ranges to the log.
2594	*
2595	* @param pVM VM Handle.
2596	* @param pHlp The info helpers.
2597	* @param pszArgs Arguments, ignored.
2598	*/
2599	static DECLCALLBACK(void) pgmR3PhysInfo(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs)
2600	{
2601	NOREF(pszArgs);
2602	pHlp->pfnPrintf(pHlp,
2603	"RAM ranges (pVM=%p)\n"
2604	"%.s %.s\n",
2605	pVM,
2606	sizeof(RTGCPHYS) * 4 + 1, "GC Phys Range ",
2607	sizeof(RTHCPTR) * 2, "pvHC ");
2608
2609	for (PPGMRAMRANGE pCur = pVM->pgm.s.pRamRangesR3; pCur; pCur = pCur->pNextR3)
2610	pHlp->pfnPrintf(pHlp,
2611	"%RGp-%RGp %RHv %s\n",
2612	pCur->GCPhys,
2613	pCur->GCPhysLast,
2614	pCur->pvR3,
2615	pCur->pszDesc);
2616	}
2617
2618	/**
2619	* Dump the page directory to the log.
2620	*
2621	* @param pVM VM Handle.
2622	* @param pHlp The info helpers.
2623	* @param pszArgs Arguments, ignored.
2624	*/
2625	static DECLCALLBACK(void) pgmR3InfoCr3(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs)
2626	{
2627	/** @todo SMP support!! */
2628	PVMCPU pVCpu = &pVM->aCpus[0];
2629
2630	/** @todo fix this! Convert the PGMR3DumpHierarchyHC functions to do guest stuff. */
2631	/* Big pages supported? */
2632	const bool fPSE = !!(CPUMGetGuestCR4(pVCpu) & X86_CR4_PSE);
2633
2634	/* Global pages supported? */
2635	const bool fPGE = !!(CPUMGetGuestCR4(pVCpu) & X86_CR4_PGE);
2636
2637	NOREF(pszArgs);
2638
2639	/*
2640	* Get page directory addresses.
2641	*/
2642	PX86PD pPDSrc = pgmGstGet32bitPDPtr(pVCpu);
2643	Assert(pPDSrc);
2644	Assert(PGMPhysGCPhys2R3PtrAssert(pVM, (RTGCPHYS)(CPUMGetGuestCR3(pVCpu) & X86_CR3_PAGE_MASK), sizeof(*pPDSrc)) == pPDSrc);
2645
2646	/*
2647	* Iterate the page directory.
2648	*/
2649	for (unsigned iPD = 0; iPD < RT_ELEMENTS(pPDSrc->a); iPD++)
2650	{
2651	X86PDE PdeSrc = pPDSrc->a[iPD];
2652	if (PdeSrc.n.u1Present)
2653	{
2654	if (PdeSrc.b.u1Size && fPSE)
2655	pHlp->pfnPrintf(pHlp,
2656	"%04X - %RGp P=%d U=%d RW=%d G=%d - BIG\n",
2657	iPD,
2658	pgmGstGet4MBPhysPage(&pVM->pgm.s, PdeSrc),
2659	PdeSrc.b.u1Present, PdeSrc.b.u1User, PdeSrc.b.u1Write, PdeSrc.b.u1Global && fPGE);
2660	else
2661	pHlp->pfnPrintf(pHlp,
2662	"%04X - %RGp P=%d U=%d RW=%d [G=%d]\n",
2663	iPD,
2664	(RTGCPHYS)(PdeSrc.u & X86_PDE_PG_MASK),
2665	PdeSrc.n.u1Present, PdeSrc.n.u1User, PdeSrc.n.u1Write, PdeSrc.b.u1Global && fPGE);
2666	}
2667	}
2668	}
2669
2670
2671	/**
2672	* Service a VMMCALLRING3_PGM_LOCK call.
2673	*
2674	* @returns VBox status code.
2675	* @param pVM The VM handle.
2676	*/
2677	VMMR3DECL(int) PGMR3LockCall(PVM pVM)
2678	{
2679	int rc = PDMR3CritSectEnterEx(&pVM->pgm.s.CritSect, true /* fHostCall */);
2680	AssertRC(rc);
2681	return rc;
2682	}
2683
2684
2685	/**
2686	* Converts a PGMMODE value to a PGM_TYPE_* \#define.
2687	*
2688	* @returns PGM_TYPE_*.
2689	* @param pgmMode The mode value to convert.
2690	*/
2691	DECLINLINE(unsigned) pgmModeToType(PGMMODE pgmMode)
2692	{
2693	switch (pgmMode)
2694	{
2695	case PGMMODE_REAL: return PGM_TYPE_REAL;
2696	case PGMMODE_PROTECTED: return PGM_TYPE_PROT;
2697	case PGMMODE_32_BIT: return PGM_TYPE_32BIT;
2698	case PGMMODE_PAE:
2699	case PGMMODE_PAE_NX: return PGM_TYPE_PAE;
2700	case PGMMODE_AMD64:
2701	case PGMMODE_AMD64_NX: return PGM_TYPE_AMD64;
2702	case PGMMODE_NESTED: return PGM_TYPE_NESTED;
2703	case PGMMODE_EPT: return PGM_TYPE_EPT;
2704	default:
2705	AssertFatalMsgFailed(("pgmMode=%d\n", pgmMode));
2706	}
2707	}
2708
2709
2710	/**
2711	* Gets the index into the paging mode data array of a SHW+GST mode.
2712	*
2713	* @returns PGM::paPagingData index.
2714	* @param uShwType The shadow paging mode type.
2715	* @param uGstType The guest paging mode type.
2716	*/
2717	DECLINLINE(unsigned) pgmModeDataIndex(unsigned uShwType, unsigned uGstType)
2718	{
2719	Assert(uShwType >= PGM_TYPE_32BIT && uShwType <= PGM_TYPE_MAX);
2720	Assert(uGstType >= PGM_TYPE_REAL && uGstType <= PGM_TYPE_AMD64);
2721	return (uShwType - PGM_TYPE_32BIT) * (PGM_TYPE_AMD64 - PGM_TYPE_REAL + 1)
2722	+ (uGstType - PGM_TYPE_REAL);
2723	}
2724
2725
2726	/**
2727	* Gets the index into the paging mode data array of a SHW+GST mode.
2728	*
2729	* @returns PGM::paPagingData index.
2730	* @param enmShw The shadow paging mode.
2731	* @param enmGst The guest paging mode.
2732	*/
2733	DECLINLINE(unsigned) pgmModeDataIndexByMode(PGMMODE enmShw, PGMMODE enmGst)
2734	{
2735	Assert(enmShw >= PGMMODE_32_BIT && enmShw <= PGMMODE_MAX);
2736	Assert(enmGst > PGMMODE_INVALID && enmGst < PGMMODE_MAX);
2737	return pgmModeDataIndex(pgmModeToType(enmShw), pgmModeToType(enmGst));
2738	}
2739
2740
2741	/**
2742	* Calculates the max data index.
2743	* @returns The number of entries in the paging data array.
2744	*/
2745	DECLINLINE(unsigned) pgmModeDataMaxIndex(void)
2746	{
2747	return pgmModeDataIndex(PGM_TYPE_MAX, PGM_TYPE_AMD64) + 1;
2748	}
2749
2750
2751	/**
2752	* Initializes the paging mode data kept in PGM::paModeData.
2753	*
2754	* @param pVM The VM handle.
2755	* @param fResolveGCAndR0 Indicate whether or not GC and Ring-0 symbols can be resolved now.
2756	* This is used early in the init process to avoid trouble with PDM
2757	* not being initialized yet.
2758	*/
2759	static int pgmR3ModeDataInit(PVM pVM, bool fResolveGCAndR0)
2760	{
2761	PPGMMODEDATA pModeData;
2762	int rc;
2763
2764	/*
2765	* Allocate the array on the first call.
2766	*/
2767	if (!pVM->pgm.s.paModeData)
2768	{
2769	pVM->pgm.s.paModeData = (PPGMMODEDATA)MMR3HeapAllocZ(pVM, MM_TAG_PGM, sizeof(PGMMODEDATA) * pgmModeDataMaxIndex());
2770	AssertReturn(pVM->pgm.s.paModeData, VERR_NO_MEMORY);
2771	}
2772
2773	/*
2774	* Initialize the array entries.
2775	*/
2776	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_32BIT, PGM_TYPE_REAL)];
2777	pModeData->uShwType = PGM_TYPE_32BIT;
2778	pModeData->uGstType = PGM_TYPE_REAL;
2779	rc = PGM_SHW_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2780	rc = PGM_GST_NAME_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2781	rc = PGM_BTH_NAME_32BIT_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2782
2783	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_32BIT, PGMMODE_PROTECTED)];
2784	pModeData->uShwType = PGM_TYPE_32BIT;
2785	pModeData->uGstType = PGM_TYPE_PROT;
2786	rc = PGM_SHW_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2787	rc = PGM_GST_NAME_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2788	rc = PGM_BTH_NAME_32BIT_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2789
2790	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_32BIT, PGM_TYPE_32BIT)];
2791	pModeData->uShwType = PGM_TYPE_32BIT;
2792	pModeData->uGstType = PGM_TYPE_32BIT;
2793	rc = PGM_SHW_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2794	rc = PGM_GST_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2795	rc = PGM_BTH_NAME_32BIT_32BIT(InitData)(pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2796
2797	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_PAE, PGM_TYPE_REAL)];
2798	pModeData->uShwType = PGM_TYPE_PAE;
2799	pModeData->uGstType = PGM_TYPE_REAL;
2800	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2801	rc = PGM_GST_NAME_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2802	rc = PGM_BTH_NAME_PAE_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2803
2804	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_PAE, PGM_TYPE_PROT)];
2805	pModeData->uShwType = PGM_TYPE_PAE;
2806	pModeData->uGstType = PGM_TYPE_PROT;
2807	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2808	rc = PGM_GST_NAME_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2809	rc = PGM_BTH_NAME_PAE_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2810
2811	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_PAE, PGM_TYPE_32BIT)];
2812	pModeData->uShwType = PGM_TYPE_PAE;
2813	pModeData->uGstType = PGM_TYPE_32BIT;
2814	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2815	rc = PGM_GST_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2816	rc = PGM_BTH_NAME_PAE_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2817
2818	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_PAE, PGM_TYPE_PAE)];
2819	pModeData->uShwType = PGM_TYPE_PAE;
2820	pModeData->uGstType = PGM_TYPE_PAE;
2821	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2822	rc = PGM_GST_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2823	rc = PGM_BTH_NAME_PAE_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2824
2825	#ifdef VBOX_WITH_64_BITS_GUESTS
2826	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_AMD64, PGM_TYPE_AMD64)];
2827	pModeData->uShwType = PGM_TYPE_AMD64;
2828	pModeData->uGstType = PGM_TYPE_AMD64;
2829	rc = PGM_SHW_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2830	rc = PGM_GST_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2831	rc = PGM_BTH_NAME_AMD64_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2832	#endif
2833
2834	/* The nested paging mode. */
2835	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_REAL)];
2836	pModeData->uShwType = PGM_TYPE_NESTED;
2837	pModeData->uGstType = PGM_TYPE_REAL;
2838	rc = PGM_GST_NAME_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2839	rc = PGM_BTH_NAME_NESTED_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2840
2841	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGMMODE_PROTECTED)];
2842	pModeData->uShwType = PGM_TYPE_NESTED;
2843	pModeData->uGstType = PGM_TYPE_PROT;
2844	rc = PGM_GST_NAME_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2845	rc = PGM_BTH_NAME_NESTED_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2846
2847	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_32BIT)];
2848	pModeData->uShwType = PGM_TYPE_NESTED;
2849	pModeData->uGstType = PGM_TYPE_32BIT;
2850	rc = PGM_GST_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2851	rc = PGM_BTH_NAME_NESTED_32BIT(InitData)(pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2852
2853	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_PAE)];
2854	pModeData->uShwType = PGM_TYPE_NESTED;
2855	pModeData->uGstType = PGM_TYPE_PAE;
2856	rc = PGM_GST_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2857	rc = PGM_BTH_NAME_NESTED_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2858
2859	#ifdef VBOX_WITH_64_BITS_GUESTS
2860	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_AMD64)];
2861	pModeData->uShwType = PGM_TYPE_NESTED;
2862	pModeData->uGstType = PGM_TYPE_AMD64;
2863	rc = PGM_GST_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2864	rc = PGM_BTH_NAME_NESTED_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2865	#endif
2866
2867	/* The shadow part of the nested callback mode depends on the host paging mode (AMD-V only). */
2868	switch (pVM->pgm.s.enmHostMode)
2869	{
2870	#if HC_ARCH_BITS == 32
2871	case SUPPAGINGMODE_32_BIT:
2872	case SUPPAGINGMODE_32_BIT_GLOBAL:
2873	for (unsigned i = PGM_TYPE_REAL; i <= PGM_TYPE_PAE; i++)
2874	{
2875	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, i)];
2876	rc = PGM_SHW_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2877	}
2878	# ifdef VBOX_WITH_64_BITS_GUESTS
2879	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_AMD64)];
2880	rc = PGM_SHW_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2881	# endif
2882	break;
2883
2884	case SUPPAGINGMODE_PAE:
2885	case SUPPAGINGMODE_PAE_NX:
2886	case SUPPAGINGMODE_PAE_GLOBAL:
2887	case SUPPAGINGMODE_PAE_GLOBAL_NX:
2888	for (unsigned i = PGM_TYPE_REAL; i <= PGM_TYPE_PAE; i++)
2889	{
2890	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, i)];
2891	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2892	}
2893	# ifdef VBOX_WITH_64_BITS_GUESTS
2894	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_AMD64)];
2895	rc = PGM_SHW_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2896	# endif
2897	break;
2898	#endif /* HC_ARCH_BITS == 32 */
2899
2900	#if HC_ARCH_BITS == 64 \|\| defined(RT_OS_DARWIN)
2901	case SUPPAGINGMODE_AMD64:
2902	case SUPPAGINGMODE_AMD64_GLOBAL:
2903	case SUPPAGINGMODE_AMD64_NX:
2904	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
2905	# ifdef VBOX_WITH_64_BITS_GUESTS
2906	for (unsigned i = PGM_TYPE_REAL; i <= PGM_TYPE_AMD64; i++)
2907	# else
2908	for (unsigned i = PGM_TYPE_REAL; i <= PGM_TYPE_PAE; i++)
2909	# endif
2910	{
2911	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, i)];
2912	rc = PGM_SHW_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2913	}
2914	break;
2915	#endif /* HC_ARCH_BITS == 64 \|\| RT_OS_DARWIN */
2916
2917	default:
2918	AssertFailed();
2919	break;
2920	}
2921
2922	/* Extended paging (EPT) / Intel VT-x */
2923	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_EPT, PGM_TYPE_REAL)];
2924	pModeData->uShwType = PGM_TYPE_EPT;
2925	pModeData->uGstType = PGM_TYPE_REAL;
2926	rc = PGM_SHW_NAME_EPT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2927	rc = PGM_GST_NAME_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2928	rc = PGM_BTH_NAME_EPT_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2929
2930	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_EPT, PGM_TYPE_PROT)];
2931	pModeData->uShwType = PGM_TYPE_EPT;
2932	pModeData->uGstType = PGM_TYPE_PROT;
2933	rc = PGM_SHW_NAME_EPT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2934	rc = PGM_GST_NAME_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2935	rc = PGM_BTH_NAME_EPT_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2936
2937	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_EPT, PGM_TYPE_32BIT)];
2938	pModeData->uShwType = PGM_TYPE_EPT;
2939	pModeData->uGstType = PGM_TYPE_32BIT;
2940	rc = PGM_SHW_NAME_EPT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2941	rc = PGM_GST_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2942	rc = PGM_BTH_NAME_EPT_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2943
2944	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_EPT, PGM_TYPE_PAE)];
2945	pModeData->uShwType = PGM_TYPE_EPT;
2946	pModeData->uGstType = PGM_TYPE_PAE;
2947	rc = PGM_SHW_NAME_EPT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2948	rc = PGM_GST_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2949	rc = PGM_BTH_NAME_EPT_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2950
2951	#ifdef VBOX_WITH_64_BITS_GUESTS
2952	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_EPT, PGM_TYPE_AMD64)];
2953	pModeData->uShwType = PGM_TYPE_EPT;
2954	pModeData->uGstType = PGM_TYPE_AMD64;
2955	rc = PGM_SHW_NAME_EPT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2956	rc = PGM_GST_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2957	rc = PGM_BTH_NAME_EPT_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2958	#endif
2959	return VINF_SUCCESS;
2960	}
2961
2962
2963	/**
2964	* Switch to different (or relocated in the relocate case) mode data.
2965	*
2966	* @param pVM The VM handle.
2967	* @param pVCpu The VMCPU to operate on.
2968	* @param enmShw The the shadow paging mode.
2969	* @param enmGst The the guest paging mode.
2970	*/
2971	static void pgmR3ModeDataSwitch(PVM pVM, PVMCPU pVCpu, PGMMODE enmShw, PGMMODE enmGst)
2972	{
2973	PPGMMODEDATA pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndexByMode(enmShw, enmGst)];
2974
2975	Assert(pModeData->uGstType == pgmModeToType(enmGst));
2976	Assert(pModeData->uShwType == pgmModeToType(enmShw));
2977
2978	/* shadow */
2979	pVCpu->pgm.s.pfnR3ShwRelocate = pModeData->pfnR3ShwRelocate;
2980	pVCpu->pgm.s.pfnR3ShwExit = pModeData->pfnR3ShwExit;
2981	pVCpu->pgm.s.pfnR3ShwGetPage = pModeData->pfnR3ShwGetPage;
2982	Assert(pVCpu->pgm.s.pfnR3ShwGetPage);
2983	pVCpu->pgm.s.pfnR3ShwModifyPage = pModeData->pfnR3ShwModifyPage;
2984
2985	pVCpu->pgm.s.pfnRCShwGetPage = pModeData->pfnRCShwGetPage;
2986	pVCpu->pgm.s.pfnRCShwModifyPage = pModeData->pfnRCShwModifyPage;
2987
2988	pVCpu->pgm.s.pfnR0ShwGetPage = pModeData->pfnR0ShwGetPage;
2989	pVCpu->pgm.s.pfnR0ShwModifyPage = pModeData->pfnR0ShwModifyPage;
2990
2991
2992	/* guest */
2993	pVCpu->pgm.s.pfnR3GstRelocate = pModeData->pfnR3GstRelocate;
2994	pVCpu->pgm.s.pfnR3GstExit = pModeData->pfnR3GstExit;
2995	pVCpu->pgm.s.pfnR3GstGetPage = pModeData->pfnR3GstGetPage;
2996	Assert(pVCpu->pgm.s.pfnR3GstGetPage);
2997	pVCpu->pgm.s.pfnR3GstModifyPage = pModeData->pfnR3GstModifyPage;
2998	pVCpu->pgm.s.pfnR3GstGetPDE = pModeData->pfnR3GstGetPDE;
2999	pVCpu->pgm.s.pfnRCGstGetPage = pModeData->pfnRCGstGetPage;
3000	pVCpu->pgm.s.pfnRCGstModifyPage = pModeData->pfnRCGstModifyPage;
3001	pVCpu->pgm.s.pfnRCGstGetPDE = pModeData->pfnRCGstGetPDE;
3002	pVCpu->pgm.s.pfnR0GstGetPage = pModeData->pfnR0GstGetPage;
3003	pVCpu->pgm.s.pfnR0GstModifyPage = pModeData->pfnR0GstModifyPage;
3004	pVCpu->pgm.s.pfnR0GstGetPDE = pModeData->pfnR0GstGetPDE;
3005
3006	/* both */
3007	pVCpu->pgm.s.pfnR3BthRelocate = pModeData->pfnR3BthRelocate;
3008	pVCpu->pgm.s.pfnR3BthInvalidatePage = pModeData->pfnR3BthInvalidatePage;
3009	pVCpu->pgm.s.pfnR3BthSyncCR3 = pModeData->pfnR3BthSyncCR3;
3010	Assert(pVCpu->pgm.s.pfnR3BthSyncCR3);
3011	pVCpu->pgm.s.pfnR3BthPrefetchPage = pModeData->pfnR3BthPrefetchPage;
3012	pVCpu->pgm.s.pfnR3BthVerifyAccessSyncPage = pModeData->pfnR3BthVerifyAccessSyncPage;
3013	#ifdef VBOX_STRICT
3014	pVCpu->pgm.s.pfnR3BthAssertCR3 = pModeData->pfnR3BthAssertCR3;
3015	#endif
3016	pVCpu->pgm.s.pfnR3BthMapCR3 = pModeData->pfnR3BthMapCR3;
3017	pVCpu->pgm.s.pfnR3BthUnmapCR3 = pModeData->pfnR3BthUnmapCR3;
3018
3019	pVCpu->pgm.s.pfnRCBthTrap0eHandler = pModeData->pfnRCBthTrap0eHandler;
3020	pVCpu->pgm.s.pfnRCBthInvalidatePage = pModeData->pfnRCBthInvalidatePage;
3021	pVCpu->pgm.s.pfnRCBthSyncCR3 = pModeData->pfnRCBthSyncCR3;
3022	pVCpu->pgm.s.pfnRCBthPrefetchPage = pModeData->pfnRCBthPrefetchPage;
3023	pVCpu->pgm.s.pfnRCBthVerifyAccessSyncPage = pModeData->pfnRCBthVerifyAccessSyncPage;
3024	#ifdef VBOX_STRICT
3025	pVCpu->pgm.s.pfnRCBthAssertCR3 = pModeData->pfnRCBthAssertCR3;
3026	#endif
3027	pVCpu->pgm.s.pfnRCBthMapCR3 = pModeData->pfnRCBthMapCR3;
3028	pVCpu->pgm.s.pfnRCBthUnmapCR3 = pModeData->pfnRCBthUnmapCR3;
3029
3030	pVCpu->pgm.s.pfnR0BthTrap0eHandler = pModeData->pfnR0BthTrap0eHandler;
3031	pVCpu->pgm.s.pfnR0BthInvalidatePage = pModeData->pfnR0BthInvalidatePage;
3032	pVCpu->pgm.s.pfnR0BthSyncCR3 = pModeData->pfnR0BthSyncCR3;
3033	pVCpu->pgm.s.pfnR0BthPrefetchPage = pModeData->pfnR0BthPrefetchPage;
3034	pVCpu->pgm.s.pfnR0BthVerifyAccessSyncPage = pModeData->pfnR0BthVerifyAccessSyncPage;
3035	#ifdef VBOX_STRICT
3036	pVCpu->pgm.s.pfnR0BthAssertCR3 = pModeData->pfnR0BthAssertCR3;
3037	#endif
3038	pVCpu->pgm.s.pfnR0BthMapCR3 = pModeData->pfnR0BthMapCR3;
3039	pVCpu->pgm.s.pfnR0BthUnmapCR3 = pModeData->pfnR0BthUnmapCR3;
3040	}
3041
3042
3043	/**
3044	* Calculates the shadow paging mode.
3045	*
3046	* @returns The shadow paging mode.
3047	* @param pVM VM handle.
3048	* @param enmGuestMode The guest mode.
3049	* @param enmHostMode The host mode.
3050	* @param enmShadowMode The current shadow mode.
3051	* @param penmSwitcher Where to store the switcher to use.
3052	* VMMSWITCHER_INVALID means no change.
3053	*/
3054	static PGMMODE pgmR3CalcShadowMode(PVM pVM, PGMMODE enmGuestMode, SUPPAGINGMODE enmHostMode, PGMMODE enmShadowMode, VMMSWITCHER *penmSwitcher)
3055	{
3056	VMMSWITCHER enmSwitcher = VMMSWITCHER_INVALID;
3057	switch (enmGuestMode)
3058	{
3059	/*
3060	* When switching to real or protected mode we don't change
3061	* anything since it's likely that we'll switch back pretty soon.
3062	*
3063	* During pgmR3InitPaging we'll end up here with PGMMODE_INVALID
3064	* and is supposed to determine which shadow paging and switcher to
3065	* use during init.
3066	*/
3067	case PGMMODE_REAL:
3068	case PGMMODE_PROTECTED:
3069	if ( enmShadowMode != PGMMODE_INVALID
3070	&& !HWACCMIsEnabled(pVM) /* always switch in hwaccm mode! */)
3071	break; /* (no change) */
3072
3073	switch (enmHostMode)
3074	{
3075	case SUPPAGINGMODE_32_BIT:
3076	case SUPPAGINGMODE_32_BIT_GLOBAL:
3077	enmShadowMode = PGMMODE_32_BIT;
3078	enmSwitcher = VMMSWITCHER_32_TO_32;
3079	break;
3080
3081	case SUPPAGINGMODE_PAE:
3082	case SUPPAGINGMODE_PAE_NX:
3083	case SUPPAGINGMODE_PAE_GLOBAL:
3084	case SUPPAGINGMODE_PAE_GLOBAL_NX:
3085	enmShadowMode = PGMMODE_PAE;
3086	enmSwitcher = VMMSWITCHER_PAE_TO_PAE;
3087	#ifdef DEBUG_bird
3088	if (RTEnvExist("VBOX_32BIT"))
3089	{
3090	enmShadowMode = PGMMODE_32_BIT;
3091	enmSwitcher = VMMSWITCHER_PAE_TO_32;
3092	}
3093	#endif
3094	break;
3095
3096	case SUPPAGINGMODE_AMD64:
3097	case SUPPAGINGMODE_AMD64_GLOBAL:
3098	case SUPPAGINGMODE_AMD64_NX:
3099	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
3100	enmShadowMode = PGMMODE_PAE;
3101	enmSwitcher = VMMSWITCHER_AMD64_TO_PAE;
3102	#ifdef DEBUG_bird
3103	if (RTEnvExist("VBOX_32BIT"))
3104	{
3105	enmShadowMode = PGMMODE_32_BIT;
3106	enmSwitcher = VMMSWITCHER_AMD64_TO_32;
3107	}
3108	#endif
3109	break;
3110
3111	default: AssertMsgFailed(("enmHostMode=%d\n", enmHostMode)); break;
3112	}
3113	break;
3114
3115	case PGMMODE_32_BIT:
3116	switch (enmHostMode)
3117	{
3118	case SUPPAGINGMODE_32_BIT:
3119	case SUPPAGINGMODE_32_BIT_GLOBAL:
3120	enmShadowMode = PGMMODE_32_BIT;
3121	enmSwitcher = VMMSWITCHER_32_TO_32;
3122	break;
3123
3124	case SUPPAGINGMODE_PAE:
3125	case SUPPAGINGMODE_PAE_NX:
3126	case SUPPAGINGMODE_PAE_GLOBAL:
3127	case SUPPAGINGMODE_PAE_GLOBAL_NX:
3128	enmShadowMode = PGMMODE_PAE;
3129	enmSwitcher = VMMSWITCHER_PAE_TO_PAE;
3130	#ifdef DEBUG_bird
3131	if (RTEnvExist("VBOX_32BIT"))
3132	{
3133	enmShadowMode = PGMMODE_32_BIT;
3134	enmSwitcher = VMMSWITCHER_PAE_TO_32;
3135	}
3136	#endif
3137	break;
3138
3139	case SUPPAGINGMODE_AMD64:
3140	case SUPPAGINGMODE_AMD64_GLOBAL:
3141	case SUPPAGINGMODE_AMD64_NX:
3142	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
3143	enmShadowMode = PGMMODE_PAE;
3144	enmSwitcher = VMMSWITCHER_AMD64_TO_PAE;
3145	#ifdef DEBUG_bird
3146	if (RTEnvExist("VBOX_32BIT"))
3147	{
3148	enmShadowMode = PGMMODE_32_BIT;
3149	enmSwitcher = VMMSWITCHER_AMD64_TO_32;
3150	}
3151	#endif
3152	break;
3153
3154	default: AssertMsgFailed(("enmHostMode=%d\n", enmHostMode)); break;
3155	}
3156	break;
3157
3158	case PGMMODE_PAE:
3159	case PGMMODE_PAE_NX: /** @todo This might require more switchers and guest+both modes. */
3160	switch (enmHostMode)
3161	{
3162	case SUPPAGINGMODE_32_BIT:
3163	case SUPPAGINGMODE_32_BIT_GLOBAL:
3164	enmShadowMode = PGMMODE_PAE;
3165	enmSwitcher = VMMSWITCHER_32_TO_PAE;
3166	break;
3167
3168	case SUPPAGINGMODE_PAE:
3169	case SUPPAGINGMODE_PAE_NX:
3170	case SUPPAGINGMODE_PAE_GLOBAL:
3171	case SUPPAGINGMODE_PAE_GLOBAL_NX:
3172	enmShadowMode = PGMMODE_PAE;
3173	enmSwitcher = VMMSWITCHER_PAE_TO_PAE;
3174	break;
3175
3176	case SUPPAGINGMODE_AMD64:
3177	case SUPPAGINGMODE_AMD64_GLOBAL:
3178	case SUPPAGINGMODE_AMD64_NX:
3179	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
3180	enmShadowMode = PGMMODE_PAE;
3181	enmSwitcher = VMMSWITCHER_AMD64_TO_PAE;
3182	break;
3183
3184	default: AssertMsgFailed(("enmHostMode=%d\n", enmHostMode)); break;
3185	}
3186	break;
3187
3188	case PGMMODE_AMD64:
3189	case PGMMODE_AMD64_NX:
3190	switch (enmHostMode)
3191	{
3192	case SUPPAGINGMODE_32_BIT:
3193	case SUPPAGINGMODE_32_BIT_GLOBAL:
3194	enmShadowMode = PGMMODE_AMD64;
3195	enmSwitcher = VMMSWITCHER_32_TO_AMD64;
3196	break;
3197
3198	case SUPPAGINGMODE_PAE:
3199	case SUPPAGINGMODE_PAE_NX:
3200	case SUPPAGINGMODE_PAE_GLOBAL:
3201	case SUPPAGINGMODE_PAE_GLOBAL_NX:
3202	enmShadowMode = PGMMODE_AMD64;
3203	enmSwitcher = VMMSWITCHER_PAE_TO_AMD64;
3204	break;
3205
3206	case SUPPAGINGMODE_AMD64:
3207	case SUPPAGINGMODE_AMD64_GLOBAL:
3208	case SUPPAGINGMODE_AMD64_NX:
3209	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
3210	enmShadowMode = PGMMODE_AMD64;
3211	enmSwitcher = VMMSWITCHER_AMD64_TO_AMD64;
3212	break;
3213
3214	default: AssertMsgFailed(("enmHostMode=%d\n", enmHostMode)); break;
3215	}
3216	break;
3217
3218
3219	default:
3220	AssertReleaseMsgFailed(("enmGuestMode=%d\n", enmGuestMode));
3221	*penmSwitcher = VMMSWITCHER_INVALID;
3222	return PGMMODE_INVALID;
3223	}
3224	/* Override the shadow mode is nested paging is active. */
3225	pVM->pgm.s.fNestedPaging = HWACCMIsNestedPagingActive(pVM);
3226	if (pVM->pgm.s.fNestedPaging)
3227	enmShadowMode = HWACCMGetShwPagingMode(pVM);
3228
3229	*penmSwitcher = enmSwitcher;
3230	return enmShadowMode;
3231	}
3232
3233
3234	/**
3235	* Performs the actual mode change.
3236	* This is called by PGMChangeMode and pgmR3InitPaging().
3237	*
3238	* @returns VBox status code. May suspend or power off the VM on error, but this
3239	* will trigger using FFs and not status codes.
3240	*
3241	* @param pVM VM handle.
3242	* @param pVCpu The VMCPU to operate on.
3243	* @param enmGuestMode The new guest mode. This is assumed to be different from
3244	* the current mode.
3245	*/
3246	VMMR3DECL(int) PGMR3ChangeMode(PVM pVM, PVMCPU pVCpu, PGMMODE enmGuestMode)
3247	{
3248	bool fIsOldGuestPagingMode64Bits = (pVCpu->pgm.s.enmGuestMode >= PGMMODE_AMD64);
3249	bool fIsNewGuestPagingMode64Bits = (enmGuestMode >= PGMMODE_AMD64);
3250
3251	Log(("PGMR3ChangeMode: Guest mode: %s -> %s\n", PGMGetModeName(pVCpu->pgm.s.enmGuestMode), PGMGetModeName(enmGuestMode)));
3252	STAM_REL_COUNTER_INC(&pVCpu->pgm.s.cGuestModeChanges);
3253
3254	/*
3255	* Calc the shadow mode and switcher.
3256	*/
3257	VMMSWITCHER enmSwitcher;
3258	PGMMODE enmShadowMode;
3259	enmShadowMode = pgmR3CalcShadowMode(pVM, enmGuestMode, pVM->pgm.s.enmHostMode, pVCpu->pgm.s.enmShadowMode, &enmSwitcher);
3260
3261	#ifdef VBOX_WITH_RAW_MODE
3262	if (enmSwitcher != VMMSWITCHER_INVALID)
3263	{
3264	/*
3265	* Select new switcher.
3266	*/
3267	int rc = VMMR3SelectSwitcher(pVM, enmSwitcher);
3268	if (RT_FAILURE(rc))
3269	{
3270	AssertReleaseMsgFailed(("VMMR3SelectSwitcher(%d) -> %Rrc\n", enmSwitcher, rc));
3271	return rc;
3272	}
3273	}
3274	#endif
3275
3276	/*
3277	* Exit old mode(s).
3278	*/
3279	#if HC_ARCH_BITS == 32
3280	/* The nested shadow paging mode for AMD-V does change when running 64 bits guests on 32 bits hosts; typically PAE <-> AMD64 */
3281	const bool fForceShwEnterExit = ( fIsOldGuestPagingMode64Bits != fIsNewGuestPagingMode64Bits
3282	&& enmShadowMode == PGMMODE_NESTED);
3283	#else
3284	const bool fForceShwEnterExit = false;
3285	#endif
3286	/* shadow */
3287	if ( enmShadowMode != pVCpu->pgm.s.enmShadowMode
3288	\|\| fForceShwEnterExit)
3289	{
3290	LogFlow(("PGMR3ChangeMode: Shadow mode: %s -> %s\n", PGMGetModeName(pVCpu->pgm.s.enmShadowMode), PGMGetModeName(enmShadowMode)));
3291	if (PGM_SHW_PFN(Exit, pVCpu))
3292	{
3293	int rc = PGM_SHW_PFN(Exit, pVCpu)(pVCpu);
3294	if (RT_FAILURE(rc))
3295	{
3296	AssertMsgFailed(("Exit failed for shadow mode %d: %Rrc\n", pVCpu->pgm.s.enmShadowMode, rc));
3297	return rc;
3298	}
3299	}
3300
3301	}
3302	else
3303	LogFlow(("PGMR3ChangeMode: Shadow mode remains: %s\n", PGMGetModeName(pVCpu->pgm.s.enmShadowMode)));
3304
3305	/* guest */
3306	if (PGM_GST_PFN(Exit, pVCpu))
3307	{
3308	int rc = PGM_GST_PFN(Exit, pVCpu)(pVCpu);
3309	if (RT_FAILURE(rc))
3310	{
3311	AssertMsgFailed(("Exit failed for guest mode %d: %Rrc\n", pVCpu->pgm.s.enmGuestMode, rc));
3312	return rc;
3313	}
3314	}
3315
3316	/*
3317	* Load new paging mode data.
3318	*/
3319	pgmR3ModeDataSwitch(pVM, pVCpu, enmShadowMode, enmGuestMode);
3320
3321	/*
3322	* Enter new shadow mode (if changed).
3323	*/
3324	if ( enmShadowMode != pVCpu->pgm.s.enmShadowMode
3325	\|\| fForceShwEnterExit)
3326	{
3327	int rc;
3328	pVCpu->pgm.s.enmShadowMode = enmShadowMode;
3329	switch (enmShadowMode)
3330	{
3331	case PGMMODE_32_BIT:
3332	rc = PGM_SHW_NAME_32BIT(Enter)(pVCpu, false);
3333	break;
3334	case PGMMODE_PAE:
3335	case PGMMODE_PAE_NX:
3336	rc = PGM_SHW_NAME_PAE(Enter)(pVCpu, false);
3337	break;
3338	case PGMMODE_AMD64:
3339	case PGMMODE_AMD64_NX:
3340	rc = PGM_SHW_NAME_AMD64(Enter)(pVCpu, fIsNewGuestPagingMode64Bits);
3341	break;
3342	case PGMMODE_NESTED:
3343	rc = PGM_SHW_NAME_NESTED(Enter)(pVCpu, fIsNewGuestPagingMode64Bits);
3344	break;
3345	case PGMMODE_EPT:
3346	rc = PGM_SHW_NAME_EPT(Enter)(pVCpu, fIsNewGuestPagingMode64Bits);
3347	break;
3348	case PGMMODE_REAL:
3349	case PGMMODE_PROTECTED:
3350	default:
3351	AssertReleaseMsgFailed(("enmShadowMode=%d\n", enmShadowMode));
3352	return VERR_INTERNAL_ERROR;
3353	}
3354	if (RT_FAILURE(rc))
3355	{
3356	AssertReleaseMsgFailed(("Entering enmShadowMode=%d failed: %Rrc\n", enmShadowMode, rc));
3357	pVCpu->pgm.s.enmShadowMode = PGMMODE_INVALID;
3358	return rc;
3359	}
3360	}
3361
3362	/*
3363	* Always flag the necessary updates
3364	*/
3365	VMCPU_FF_SET(pVCpu, VMCPU_FF_PGM_SYNC_CR3);
3366
3367	/*
3368	* Enter the new guest and shadow+guest modes.
3369	*/
3370	int rc = -1;
3371	int rc2 = -1;
3372	RTGCPHYS GCPhysCR3 = NIL_RTGCPHYS;
3373	pVCpu->pgm.s.enmGuestMode = enmGuestMode;
3374	switch (enmGuestMode)
3375	{
3376	case PGMMODE_REAL:
3377	rc = PGM_GST_NAME_REAL(Enter)(pVCpu, NIL_RTGCPHYS);
3378	switch (pVCpu->pgm.s.enmShadowMode)
3379	{
3380	case PGMMODE_32_BIT:
3381	rc2 = PGM_BTH_NAME_32BIT_REAL(Enter)(pVCpu, NIL_RTGCPHYS);
3382	break;
3383	case PGMMODE_PAE:
3384	case PGMMODE_PAE_NX:
3385	rc2 = PGM_BTH_NAME_PAE_REAL(Enter)(pVCpu, NIL_RTGCPHYS);
3386	break;
3387	case PGMMODE_NESTED:
3388	rc2 = PGM_BTH_NAME_NESTED_REAL(Enter)(pVCpu, NIL_RTGCPHYS);
3389	break;
3390	case PGMMODE_EPT:
3391	rc2 = PGM_BTH_NAME_EPT_REAL(Enter)(pVCpu, NIL_RTGCPHYS);
3392	break;
3393	case PGMMODE_AMD64:
3394	case PGMMODE_AMD64_NX:
3395	AssertMsgFailed(("Should use PAE shadow mode!\n"));
3396	default: AssertFailed(); break;
3397	}
3398	break;
3399
3400	case PGMMODE_PROTECTED:
3401	rc = PGM_GST_NAME_PROT(Enter)(pVCpu, NIL_RTGCPHYS);
3402	switch (pVCpu->pgm.s.enmShadowMode)
3403	{
3404	case PGMMODE_32_BIT:
3405	rc2 = PGM_BTH_NAME_32BIT_PROT(Enter)(pVCpu, NIL_RTGCPHYS);
3406	break;
3407	case PGMMODE_PAE:
3408	case PGMMODE_PAE_NX:
3409	rc2 = PGM_BTH_NAME_PAE_PROT(Enter)(pVCpu, NIL_RTGCPHYS);
3410	break;
3411	case PGMMODE_NESTED:
3412	rc2 = PGM_BTH_NAME_NESTED_PROT(Enter)(pVCpu, NIL_RTGCPHYS);
3413	break;
3414	case PGMMODE_EPT:
3415	rc2 = PGM_BTH_NAME_EPT_PROT(Enter)(pVCpu, NIL_RTGCPHYS);
3416	break;
3417	case PGMMODE_AMD64:
3418	case PGMMODE_AMD64_NX:
3419	AssertMsgFailed(("Should use PAE shadow mode!\n"));
3420	default: AssertFailed(); break;
3421	}
3422	break;
3423
3424	case PGMMODE_32_BIT:
3425	GCPhysCR3 = CPUMGetGuestCR3(pVCpu) & X86_CR3_PAGE_MASK;
3426	rc = PGM_GST_NAME_32BIT(Enter)(pVCpu, GCPhysCR3);
3427	switch (pVCpu->pgm.s.enmShadowMode)
3428	{
3429	case PGMMODE_32_BIT:
3430	rc2 = PGM_BTH_NAME_32BIT_32BIT(Enter)(pVCpu, GCPhysCR3);
3431	break;
3432	case PGMMODE_PAE:
3433	case PGMMODE_PAE_NX:
3434	rc2 = PGM_BTH_NAME_PAE_32BIT(Enter)(pVCpu, GCPhysCR3);
3435	break;
3436	case PGMMODE_NESTED:
3437	rc2 = PGM_BTH_NAME_NESTED_32BIT(Enter)(pVCpu, GCPhysCR3);
3438	break;
3439	case PGMMODE_EPT:
3440	rc2 = PGM_BTH_NAME_EPT_32BIT(Enter)(pVCpu, GCPhysCR3);
3441	break;
3442	case PGMMODE_AMD64:
3443	case PGMMODE_AMD64_NX:
3444	AssertMsgFailed(("Should use PAE shadow mode!\n"));
3445	default: AssertFailed(); break;
3446	}
3447	break;
3448
3449	case PGMMODE_PAE_NX:
3450	case PGMMODE_PAE:
3451	{
3452	uint32_t u32Dummy, u32Features;
3453
3454	CPUMGetGuestCpuId(pVCpu, 1, &u32Dummy, &u32Dummy, &u32Dummy, &u32Features);
3455	if (!(u32Features & X86_CPUID_FEATURE_EDX_PAE))
3456	return VMSetRuntimeError(pVM, VMSETRTERR_FLAGS_FATAL, "PAEmode",
3457	N_("The guest is trying to switch to the PAE mode which is currently disabled by default in VirtualBox. PAE support can be enabled using the VM settings (General/Advanced)"));
3458
3459	GCPhysCR3 = CPUMGetGuestCR3(pVCpu) & X86_CR3_PAE_PAGE_MASK;
3460	rc = PGM_GST_NAME_PAE(Enter)(pVCpu, GCPhysCR3);
3461	switch (pVCpu->pgm.s.enmShadowMode)
3462	{
3463	case PGMMODE_PAE:
3464	case PGMMODE_PAE_NX:
3465	rc2 = PGM_BTH_NAME_PAE_PAE(Enter)(pVCpu, GCPhysCR3);
3466	break;
3467	case PGMMODE_NESTED:
3468	rc2 = PGM_BTH_NAME_NESTED_PAE(Enter)(pVCpu, GCPhysCR3);
3469	break;
3470	case PGMMODE_EPT:
3471	rc2 = PGM_BTH_NAME_EPT_PAE(Enter)(pVCpu, GCPhysCR3);
3472	break;
3473	case PGMMODE_32_BIT:
3474	case PGMMODE_AMD64:
3475	case PGMMODE_AMD64_NX:
3476	AssertMsgFailed(("Should use PAE shadow mode!\n"));
3477	default: AssertFailed(); break;
3478	}
3479	break;
3480	}
3481
3482	#ifdef VBOX_WITH_64_BITS_GUESTS
3483	case PGMMODE_AMD64_NX:
3484	case PGMMODE_AMD64:
3485	GCPhysCR3 = CPUMGetGuestCR3(pVCpu) & UINT64_C(0xfffffffffffff000); /** @todo define this mask! */
3486	rc = PGM_GST_NAME_AMD64(Enter)(pVCpu, GCPhysCR3);
3487	switch (pVCpu->pgm.s.enmShadowMode)
3488	{
3489	case PGMMODE_AMD64:
3490	case PGMMODE_AMD64_NX:
3491	rc2 = PGM_BTH_NAME_AMD64_AMD64(Enter)(pVCpu, GCPhysCR3);
3492	break;
3493	case PGMMODE_NESTED:
3494	rc2 = PGM_BTH_NAME_NESTED_AMD64(Enter)(pVCpu, GCPhysCR3);
3495	break;
3496	case PGMMODE_EPT:
3497	rc2 = PGM_BTH_NAME_EPT_AMD64(Enter)(pVCpu, GCPhysCR3);
3498	break;
3499	case PGMMODE_32_BIT:
3500	case PGMMODE_PAE:
3501	case PGMMODE_PAE_NX:
3502	AssertMsgFailed(("Should use AMD64 shadow mode!\n"));
3503	default: AssertFailed(); break;
3504	}
3505	break;
3506	#endif
3507
3508	default:
3509	AssertReleaseMsgFailed(("enmGuestMode=%d\n", enmGuestMode));
3510	rc = VERR_NOT_IMPLEMENTED;
3511	break;
3512	}
3513
3514	/* status codes. */
3515	AssertRC(rc);
3516	AssertRC(rc2);
3517	if (RT_SUCCESS(rc))
3518	{
3519	rc = rc2;
3520	if (RT_SUCCESS(rc)) /* no informational status codes. */
3521	rc = VINF_SUCCESS;
3522	}
3523
3524	/* Notify HWACCM as well. */
3525	HWACCMR3PagingModeChanged(pVM, pVCpu, pVCpu->pgm.s.enmShadowMode, pVCpu->pgm.s.enmGuestMode);
3526	return rc;
3527	}
3528
3529
3530	/**
3531	* Called by pgmPoolFlushAllInt prior to flushing the pool.
3532	*
3533	* @returns VBox status code, fully asserted.
3534	* @param pVM The VM handle.
3535	* @param pVCpu The VMCPU to operate on.
3536	*/
3537	int pgmR3ExitShadowModeBeforePoolFlush(PVM pVM, PVMCPU pVCpu)
3538	{
3539	/* Unmap the old CR3 value before flushing everything. */
3540	int rc = PGM_BTH_PFN(UnmapCR3, pVCpu)(pVCpu);
3541	AssertRC(rc);
3542
3543	/* Exit the current shadow paging mode as well; nested paging and EPT use a root CR3 which will get flushed here. */
3544	rc = PGM_SHW_PFN(Exit, pVCpu)(pVCpu);
3545	AssertRC(rc);
3546	Assert(pVCpu->pgm.s.pShwPageCR3R3 == NULL);
3547	return rc;
3548	}
3549
3550
3551	/**
3552	* Called by pgmPoolFlushAllInt after flushing the pool.
3553	*
3554	* @returns VBox status code, fully asserted.
3555	* @param pVM The VM handle.
3556	* @param pVCpu The VMCPU to operate on.
3557	*/
3558	int pgmR3ReEnterShadowModeAfterPoolFlush(PVM pVM, PVMCPU pVCpu)
3559	{
3560	pVCpu->pgm.s.enmShadowMode = PGMMODE_INVALID;
3561	int rc = PGMR3ChangeMode(pVM, pVCpu, PGMGetGuestMode(pVCpu));
3562	Assert(VMCPU_FF_ISSET(pVCpu, VMCPU_FF_PGM_SYNC_CR3));
3563	AssertRCReturn(rc, rc);
3564	AssertRCSuccessReturn(rc, VERR_IPE_UNEXPECTED_INFO_STATUS);
3565
3566	Assert(pVCpu->pgm.s.pShwPageCR3R3 != NULL);
3567	AssertMsg( pVCpu->pgm.s.enmShadowMode >= PGMMODE_NESTED
3568	\|\| CPUMGetHyperCR3(pVCpu) == PGMGetHyperCR3(pVCpu),
3569	("%RHp != %RHp %s\n", (RTHCPHYS)CPUMGetHyperCR3(pVCpu), PGMGetHyperCR3(pVCpu), PGMGetModeName(pVCpu->pgm.s.enmShadowMode)));
3570	return rc;
3571	}
3572
3573	#ifdef VBOX_WITH_DEBUGGER
3574
3575	/**
3576	* The '.pgmram' command.
3577	*
3578	* @returns VBox status.
3579	* @param pCmd Pointer to the command descriptor (as registered).
3580	* @param pCmdHlp Pointer to command helper functions.
3581	* @param pVM Pointer to the current VM (if any).
3582	* @param paArgs Pointer to (readonly) array of arguments.
3583	* @param cArgs Number of arguments in the array.
3584	*/
3585	static DECLCALLBACK(int) pgmR3CmdRam(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs)
3586	{
3587	/*
3588	* Validate input.
3589	*/
3590	if (!pVM)
3591	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires a VM to be selected.\n");
3592	if (!pVM->pgm.s.pRamRangesRC)
3593	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Sorry, no Ram is registered.\n");
3594
3595	/*
3596	* Dump the ranges.
3597	*/
3598	int rc = pCmdHlp->pfnPrintf(pCmdHlp, NULL, "From - To (incl) pvHC\n");
3599	PPGMRAMRANGE pRam;
3600	for (pRam = pVM->pgm.s.pRamRangesR3; pRam; pRam = pRam->pNextR3)
3601	{
3602	rc = pCmdHlp->pfnPrintf(pCmdHlp, NULL,
3603	"%RGp - %RGp %p\n",
3604	pRam->GCPhys, pRam->GCPhysLast, pRam->pvR3);
3605	if (RT_FAILURE(rc))
3606	return rc;
3607	}
3608
3609	return VINF_SUCCESS;
3610	}
3611
3612
3613	/**
3614	* The '.pgmerror' and '.pgmerroroff' commands.
3615	*
3616	* @returns VBox status.
3617	* @param pCmd Pointer to the command descriptor (as registered).
3618	* @param pCmdHlp Pointer to command helper functions.
3619	* @param pVM Pointer to the current VM (if any).
3620	* @param paArgs Pointer to (readonly) array of arguments.
3621	* @param cArgs Number of arguments in the array.
3622	*/
3623	static DECLCALLBACK(int) pgmR3CmdError(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs)
3624	{
3625	/*
3626	* Validate input.
3627	*/
3628	if (!pVM)
3629	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires a VM to be selected.\n");
3630	AssertReturn(cArgs == 0 \|\| (cArgs == 1 && paArgs[0].enmType == DBGCVAR_TYPE_STRING),
3631	pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: Hit bug in the parser.\n"));
3632
3633	if (!cArgs)
3634	{
3635	/*
3636	* Print the list of error injection locations with status.
3637	*/
3638	pCmdHlp->pfnPrintf(pCmdHlp, NULL, "PGM error inject locations:\n");
3639	pCmdHlp->pfnPrintf(pCmdHlp, NULL, " handy - %RTbool\n", pVM->pgm.s.fErrInjHandyPages);
3640	}
3641	else
3642	{
3643
3644	/*
3645	* String switch on where to inject the error.
3646	*/
3647	bool const fNewState = !strcmp(pCmd->pszCmd, "pgmerror");
3648	const char *pszWhere = paArgs[0].u.pszString;
3649	if (!strcmp(pszWhere, "handy"))
3650	ASMAtomicWriteBool(&pVM->pgm.s.fErrInjHandyPages, fNewState);
3651	else
3652	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: Invalid 'where' value: %s.\n", pszWhere);
3653	pCmdHlp->pfnPrintf(pCmdHlp, NULL, "done\n");
3654	}
3655	return VINF_SUCCESS;
3656	}
3657
3658
3659	/**
3660	* The '.pgmsync' command.
3661	*
3662	* @returns VBox status.
3663	* @param pCmd Pointer to the command descriptor (as registered).
3664	* @param pCmdHlp Pointer to command helper functions.
3665	* @param pVM Pointer to the current VM (if any).
3666	* @param paArgs Pointer to (readonly) array of arguments.
3667	* @param cArgs Number of arguments in the array.
3668	*/
3669	static DECLCALLBACK(int) pgmR3CmdSync(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs)
3670	{
3671	/** @todo SMP support */
3672
3673	/*
3674	* Validate input.
3675	*/
3676	if (!pVM)
3677	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires a VM to be selected.\n");
3678
3679	PVMCPU pVCpu = &pVM->aCpus[0];
3680
3681	/*
3682	* Force page directory sync.
3683	*/
3684	VMCPU_FF_SET(pVCpu, VMCPU_FF_PGM_SYNC_CR3);
3685
3686	int rc = pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Forcing page directory sync.\n");
3687	if (RT_FAILURE(rc))
3688	return rc;
3689
3690	return VINF_SUCCESS;
3691	}
3692
3693
3694	#ifdef VBOX_STRICT
3695	/**
3696	* The '.pgmassertcr3' command.
3697	*
3698	* @returns VBox status.
3699	* @param pCmd Pointer to the command descriptor (as registered).
3700	* @param pCmdHlp Pointer to command helper functions.
3701	* @param pVM Pointer to the current VM (if any).
3702	* @param paArgs Pointer to (readonly) array of arguments.
3703	* @param cArgs Number of arguments in the array.
3704	*/
3705	static DECLCALLBACK(int) pgmR3CmdAssertCR3(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs)
3706	{
3707	/** @todo SMP support!! */
3708
3709	/*
3710	* Validate input.
3711	*/
3712	if (!pVM)
3713	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires a VM to be selected.\n");
3714
3715	PVMCPU pVCpu = &pVM->aCpus[0];
3716
3717	int rc = pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Checking shadow CR3 page tables for consistency.\n");
3718	if (RT_FAILURE(rc))
3719	return rc;
3720
3721	PGMAssertCR3(pVM, pVCpu, CPUMGetGuestCR3(pVCpu), CPUMGetGuestCR4(pVCpu));
3722
3723	return VINF_SUCCESS;
3724	}
3725	#endif /* VBOX_STRICT */
3726
3727
3728	/**
3729	* The '.pgmsyncalways' command.
3730	*
3731	* @returns VBox status.
3732	* @param pCmd Pointer to the command descriptor (as registered).
3733	* @param pCmdHlp Pointer to command helper functions.
3734	* @param pVM Pointer to the current VM (if any).
3735	* @param paArgs Pointer to (readonly) array of arguments.
3736	* @param cArgs Number of arguments in the array.
3737	*/
3738	static DECLCALLBACK(int) pgmR3CmdSyncAlways(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs)
3739	{
3740	/** @todo SMP support!! */
3741	PVMCPU pVCpu = &pVM->aCpus[0];
3742
3743	/*
3744	* Validate input.
3745	*/
3746	if (!pVM)
3747	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires a VM to be selected.\n");
3748
3749	/*
3750	* Force page directory sync.
3751	*/
3752	if (pVCpu->pgm.s.fSyncFlags & PGM_SYNC_ALWAYS)
3753	{
3754	ASMAtomicAndU32(&pVCpu->pgm.s.fSyncFlags, ~PGM_SYNC_ALWAYS);
3755	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Disabled permanent forced page directory syncing.\n");
3756	}
3757	else
3758	{
3759	ASMAtomicOrU32(&pVCpu->pgm.s.fSyncFlags, PGM_SYNC_ALWAYS);
3760	VMCPU_FF_SET(pVCpu, VMCPU_FF_PGM_SYNC_CR3);
3761	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Enabled permanent forced page directory syncing.\n");
3762	}
3763	}
3764
3765
3766	/**
3767	* The '.pgmphystofile' command.
3768	*
3769	* @returns VBox status.
3770	* @param pCmd Pointer to the command descriptor (as registered).
3771	* @param pCmdHlp Pointer to command helper functions.
3772	* @param pVM Pointer to the current VM (if any).
3773	* @param paArgs Pointer to (readonly) array of arguments.
3774	* @param cArgs Number of arguments in the array.
3775	*/
3776	static DECLCALLBACK(int) pgmR3CmdPhysToFile(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs)
3777	{
3778	/*
3779	* Validate input.
3780	*/
3781	if (!pVM)
3782	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires a VM to be selected.\n");
3783	if ( cArgs < 1
3784	\|\| cArgs > 2
3785	\|\| paArgs[0].enmType != DBGCVAR_TYPE_STRING
3786	\|\| ( cArgs > 1
3787	&& paArgs[1].enmType != DBGCVAR_TYPE_STRING))
3788	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: parser error, invalid arguments.\n");
3789	if ( cArgs >= 2
3790	&& strcmp(paArgs[1].u.pszString, "nozero"))
3791	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: Invalid 2nd argument '%s', must be 'nozero'.\n", paArgs[1].u.pszString);
3792	bool fIncZeroPgs = cArgs < 2;
3793
3794	/*
3795	* Open the output file and get the ram parameters.
3796	*/
3797	RTFILE hFile;
3798	int rc = RTFileOpen(&hFile, paArgs[0].u.pszString, RTFILE_O_WRITE \| RTFILE_O_CREATE_REPLACE \| RTFILE_O_DENY_WRITE);
3799	if (RT_FAILURE(rc))
3800	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: RTFileOpen(,'%s',) -> %Rrc.\n", paArgs[0].u.pszString, rc);
3801
3802	uint32_t cbRamHole = 0;
3803	CFGMR3QueryU32Def(CFGMR3GetRoot(pVM), "RamHoleSize", &cbRamHole, MM_RAM_HOLE_SIZE_DEFAULT);
3804	uint64_t cbRam = 0;
3805	CFGMR3QueryU64Def(CFGMR3GetRoot(pVM), "RamSize", &cbRam, 0);
3806	RTGCPHYS GCPhysEnd = cbRam + cbRamHole;
3807
3808	/*
3809	* Dump the physical memory, page by page.
3810	*/
3811	RTGCPHYS GCPhys = 0;
3812	char abZeroPg[PAGE_SIZE];
3813	RT_ZERO(abZeroPg);
3814
3815	pgmLock(pVM);
3816	for (PPGMRAMRANGE pRam = pVM->pgm.s.pRamRangesR3;
3817	pRam && pRam->GCPhys < GCPhysEnd && RT_SUCCESS(rc);
3818	pRam = pRam->pNextR3)
3819	{
3820	/* fill the gap */
3821	if (pRam->GCPhys > GCPhys && fIncZeroPgs)
3822	{
3823	while (pRam->GCPhys > GCPhys && RT_SUCCESS(rc))
3824	{
3825	rc = RTFileWrite(hFile, abZeroPg, PAGE_SIZE, NULL);
3826	GCPhys += PAGE_SIZE;
3827	}
3828	}
3829
3830	PCPGMPAGE pPage = &pRam->aPages[0];
3831	while (GCPhys < pRam->GCPhysLast && RT_SUCCESS(rc))
3832	{
3833	if ( PGM_PAGE_IS_ZERO(pPage)
3834	\|\| PGM_PAGE_IS_BALLOONED(pPage))
3835	{
3836	if (fIncZeroPgs)
3837	{
3838	rc = RTFileWrite(hFile, abZeroPg, PAGE_SIZE, NULL);
3839	if (RT_FAILURE(rc))
3840	pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: RTFileWrite -> %Rrc at GCPhys=%RGp.\n", rc, GCPhys);
3841	}
3842	}
3843	else
3844	{
3845	switch (PGM_PAGE_GET_TYPE(pPage))
3846	{
3847	case PGMPAGETYPE_RAM:
3848	case PGMPAGETYPE_ROM_SHADOW: /* trouble?? */
3849	case PGMPAGETYPE_ROM:
3850	case PGMPAGETYPE_MMIO2:
3851	{
3852	void const *pvPage;
3853	PGMPAGEMAPLOCK Lock;
3854	rc = PGMPhysGCPhys2CCPtrReadOnly(pVM, GCPhys, &pvPage, &Lock);
3855	if (RT_SUCCESS(rc))
3856	{
3857	rc = RTFileWrite(hFile, pvPage, PAGE_SIZE, NULL);
3858	PGMPhysReleasePageMappingLock(pVM, &Lock);
3859	if (RT_FAILURE(rc))
3860	pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: RTFileWrite -> %Rrc at GCPhys=%RGp.\n", rc, GCPhys);
3861	}
3862	else
3863	pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: PGMPhysGCPhys2CCPtrReadOnly -> %Rrc at GCPhys=%RGp.\n", rc, GCPhys);
3864	break;
3865	}
3866
3867	default:
3868	AssertFailed();
3869	case PGMPAGETYPE_MMIO2_ALIAS_MMIO:
3870	case PGMPAGETYPE_MMIO:
3871	if (fIncZeroPgs)
3872	{
3873	rc = RTFileWrite(hFile, abZeroPg, PAGE_SIZE, NULL);
3874	if (RT_FAILURE(rc))
3875	pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: RTFileWrite -> %Rrc at GCPhys=%RGp.\n", rc, GCPhys);
3876	}
3877	break;
3878	}
3879	}
3880
3881
3882	/* advance */
3883	GCPhys += PAGE_SIZE;
3884	pPage++;
3885	}
3886	}
3887	pgmUnlock(pVM);
3888
3889	RTFileClose(hFile);
3890	if (RT_SUCCESS(rc))
3891	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Successfully saved physical memory to '%s'.\n", paArgs[0].u.pszString);
3892	return VINF_SUCCESS;
3893	}
3894
3895	#endif /* VBOX_WITH_DEBUGGER */
3896
3897	/**
3898	* pvUser argument of the pgmR3CheckIntegrity*Node callbacks.
3899	*/
3900	typedef struct PGMCHECKINTARGS
3901	{
3902	bool fLeftToRight; /*< true: left-to-right; false: right-to-left. /
3903	PPGMPHYSHANDLER pPrevPhys;
3904	PPGMVIRTHANDLER pPrevVirt;
3905	PPGMPHYS2VIRTHANDLER pPrevPhys2Virt;
3906	PVM pVM;
3907	} PGMCHECKINTARGS, *PPGMCHECKINTARGS;
3908
3909	/**
3910	* Validate a node in the physical handler tree.
3911	*
3912	* @returns 0 on if ok, other wise 1.
3913	* @param pNode The handler node.
3914	* @param pvUser pVM.
3915	*/
3916	static DECLCALLBACK(int) pgmR3CheckIntegrityPhysHandlerNode(PAVLROGCPHYSNODECORE pNode, void *pvUser)
3917	{
3918	PPGMCHECKINTARGS pArgs = (PPGMCHECKINTARGS)pvUser;
3919	PPGMPHYSHANDLER pCur = (PPGMPHYSHANDLER)pNode;
3920	AssertReleaseReturn(!((uintptr_t)pCur & 7), 1);
3921	AssertReleaseMsg(pCur->Core.Key <= pCur->Core.KeyLast,("pCur=%p %RGp-%RGp %s\n", pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc));
3922	AssertReleaseMsg( !pArgs->pPrevPhys
3923	\|\| (pArgs->fLeftToRight ? pArgs->pPrevPhys->Core.KeyLast < pCur->Core.Key : pArgs->pPrevPhys->Core.KeyLast > pCur->Core.Key),
3924	("pPrevPhys=%p %RGp-%RGp %s\n"
3925	" pCur=%p %RGp-%RGp %s\n",
3926	pArgs->pPrevPhys, pArgs->pPrevPhys->Core.Key, pArgs->pPrevPhys->Core.KeyLast, pArgs->pPrevPhys->pszDesc,
3927	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc));
3928	pArgs->pPrevPhys = pCur;
3929	return 0;
3930	}
3931
3932
3933	/**
3934	* Validate a node in the virtual handler tree.
3935	*
3936	* @returns 0 on if ok, other wise 1.
3937	* @param pNode The handler node.
3938	* @param pvUser pVM.
3939	*/
3940	static DECLCALLBACK(int) pgmR3CheckIntegrityVirtHandlerNode(PAVLROGCPTRNODECORE pNode, void *pvUser)
3941	{
3942	PPGMCHECKINTARGS pArgs = (PPGMCHECKINTARGS)pvUser;
3943	PPGMVIRTHANDLER pCur = (PPGMVIRTHANDLER)pNode;
3944	AssertReleaseReturn(!((uintptr_t)pCur & 7), 1);
3945	AssertReleaseMsg(pCur->Core.Key <= pCur->Core.KeyLast,("pCur=%p %RGv-%RGv %s\n", pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc));
3946	AssertReleaseMsg( !pArgs->pPrevVirt
3947	\|\| (pArgs->fLeftToRight ? pArgs->pPrevVirt->Core.KeyLast < pCur->Core.Key : pArgs->pPrevVirt->Core.KeyLast > pCur->Core.Key),
3948	("pPrevVirt=%p %RGv-%RGv %s\n"
3949	" pCur=%p %RGv-%RGv %s\n",
3950	pArgs->pPrevVirt, pArgs->pPrevVirt->Core.Key, pArgs->pPrevVirt->Core.KeyLast, pArgs->pPrevVirt->pszDesc,
3951	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc));
3952	for (unsigned iPage = 0; iPage < pCur->cPages; iPage++)
3953	{
3954	AssertReleaseMsg(pCur->aPhysToVirt[iPage].offVirtHandler == -RT_OFFSETOF(PGMVIRTHANDLER, aPhysToVirt[iPage]),
3955	("pCur=%p %RGv-%RGv %s\n"
3956	"iPage=%d offVirtHandle=%#x expected %#x\n",
3957	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc,
3958	iPage, pCur->aPhysToVirt[iPage].offVirtHandler, -RT_OFFSETOF(PGMVIRTHANDLER, aPhysToVirt[iPage])));
3959	}
3960	pArgs->pPrevVirt = pCur;
3961	return 0;
3962	}
3963
3964
3965	/**
3966	* Validate a node in the virtual handler tree.
3967	*
3968	* @returns 0 on if ok, other wise 1.
3969	* @param pNode The handler node.
3970	* @param pvUser pVM.
3971	*/
3972	static DECLCALLBACK(int) pgmR3CheckIntegrityPhysToVirtHandlerNode(PAVLROGCPHYSNODECORE pNode, void *pvUser)
3973	{
3974	PPGMCHECKINTARGS pArgs = (PPGMCHECKINTARGS)pvUser;
3975	PPGMPHYS2VIRTHANDLER pCur = (PPGMPHYS2VIRTHANDLER)pNode;
3976	AssertReleaseMsgReturn(!((uintptr_t)pCur & 3), ("\n"), 1);
3977	AssertReleaseMsgReturn(!(pCur->offVirtHandler & 3), ("\n"), 1);
3978	AssertReleaseMsg(pCur->Core.Key <= pCur->Core.KeyLast,("pCur=%p %RGp-%RGp\n", pCur, pCur->Core.Key, pCur->Core.KeyLast));
3979	AssertReleaseMsg( !pArgs->pPrevPhys2Virt
3980	\|\| (pArgs->fLeftToRight ? pArgs->pPrevPhys2Virt->Core.KeyLast < pCur->Core.Key : pArgs->pPrevPhys2Virt->Core.KeyLast > pCur->Core.Key),
3981	("pPrevPhys2Virt=%p %RGp-%RGp\n"
3982	" pCur=%p %RGp-%RGp\n",
3983	pArgs->pPrevPhys2Virt, pArgs->pPrevPhys2Virt->Core.Key, pArgs->pPrevPhys2Virt->Core.KeyLast,
3984	pCur, pCur->Core.Key, pCur->Core.KeyLast));
3985	AssertReleaseMsg( !pArgs->pPrevPhys2Virt
3986	\|\| (pArgs->fLeftToRight ? pArgs->pPrevPhys2Virt->Core.KeyLast < pCur->Core.Key : pArgs->pPrevPhys2Virt->Core.KeyLast > pCur->Core.Key),
3987	("pPrevPhys2Virt=%p %RGp-%RGp\n"
3988	" pCur=%p %RGp-%RGp\n",
3989	pArgs->pPrevPhys2Virt, pArgs->pPrevPhys2Virt->Core.Key, pArgs->pPrevPhys2Virt->Core.KeyLast,
3990	pCur, pCur->Core.Key, pCur->Core.KeyLast));
3991	AssertReleaseMsg((pCur->offNextAlias & (PGMPHYS2VIRTHANDLER_IN_TREE \| PGMPHYS2VIRTHANDLER_IS_HEAD)) == (PGMPHYS2VIRTHANDLER_IN_TREE \| PGMPHYS2VIRTHANDLER_IS_HEAD),
3992	("pCur=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
3993	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias));
3994	if (pCur->offNextAlias & PGMPHYS2VIRTHANDLER_OFF_MASK)
3995	{
3996	PPGMPHYS2VIRTHANDLER pCur2 = pCur;
3997	for (;;)
3998	{
3999	pCur2 = (PPGMPHYS2VIRTHANDLER)((intptr_t)pCur + (pCur->offNextAlias & PGMPHYS2VIRTHANDLER_OFF_MASK));
4000	AssertReleaseMsg(pCur2 != pCur,
4001	(" pCur=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
4002	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias));
4003	AssertReleaseMsg((pCur2->offNextAlias & (PGMPHYS2VIRTHANDLER_IN_TREE \| PGMPHYS2VIRTHANDLER_IS_HEAD)) == PGMPHYS2VIRTHANDLER_IN_TREE,
4004	(" pCur=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n"
4005	"pCur2=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
4006	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias,
4007	pCur2, pCur2->Core.Key, pCur2->Core.KeyLast, pCur2->offVirtHandler, pCur2->offNextAlias));
4008	AssertReleaseMsg((pCur2->Core.Key ^ pCur->Core.Key) < PAGE_SIZE,
4009	(" pCur=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n"
4010	"pCur2=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
4011	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias,
4012	pCur2, pCur2->Core.Key, pCur2->Core.KeyLast, pCur2->offVirtHandler, pCur2->offNextAlias));
4013	AssertReleaseMsg((pCur2->Core.KeyLast ^ pCur->Core.KeyLast) < PAGE_SIZE,
4014	(" pCur=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n"
4015	"pCur2=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
4016	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias,
4017	pCur2, pCur2->Core.Key, pCur2->Core.KeyLast, pCur2->offVirtHandler, pCur2->offNextAlias));
4018	if (!(pCur2->offNextAlias & PGMPHYS2VIRTHANDLER_OFF_MASK))
4019	break;
4020	}
4021	}
4022
4023	pArgs->pPrevPhys2Virt = pCur;
4024	return 0;
4025	}
4026
4027
4028	/**
4029	* Perform an integrity check on the PGM component.
4030	*
4031	* @returns VINF_SUCCESS if everything is fine.
4032	* @returns VBox error status after asserting on integrity breach.
4033	* @param pVM The VM handle.
4034	*/
4035	VMMR3DECL(int) PGMR3CheckIntegrity(PVM pVM)
4036	{
4037	AssertReleaseReturn(pVM->pgm.s.offVM, VERR_INTERNAL_ERROR);
4038
4039	/*
4040	* Check the trees.
4041	*/
4042	int cErrors = 0;
4043	const static PGMCHECKINTARGS s_LeftToRight = { true, NULL, NULL, NULL, pVM };
4044	const static PGMCHECKINTARGS s_RightToLeft = { false, NULL, NULL, NULL, pVM };
4045	PGMCHECKINTARGS Args = s_LeftToRight;
4046	cErrors += RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesR3->PhysHandlers, true, pgmR3CheckIntegrityPhysHandlerNode, &Args);
4047	Args = s_RightToLeft;
4048	cErrors += RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesR3->PhysHandlers, false, pgmR3CheckIntegrityPhysHandlerNode, &Args);
4049	Args = s_LeftToRight;
4050	cErrors += RTAvlroGCPtrDoWithAll( &pVM->pgm.s.pTreesR3->VirtHandlers, true, pgmR3CheckIntegrityVirtHandlerNode, &Args);
4051	Args = s_RightToLeft;
4052	cErrors += RTAvlroGCPtrDoWithAll( &pVM->pgm.s.pTreesR3->VirtHandlers, false, pgmR3CheckIntegrityVirtHandlerNode, &Args);
4053	Args = s_LeftToRight;
4054	cErrors += RTAvlroGCPtrDoWithAll( &pVM->pgm.s.pTreesR3->HyperVirtHandlers, true, pgmR3CheckIntegrityVirtHandlerNode, &Args);
4055	Args = s_RightToLeft;
4056	cErrors += RTAvlroGCPtrDoWithAll( &pVM->pgm.s.pTreesR3->HyperVirtHandlers, false, pgmR3CheckIntegrityVirtHandlerNode, &Args);
4057	Args = s_LeftToRight;
4058	cErrors += RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesR3->PhysToVirtHandlers, true, pgmR3CheckIntegrityPhysToVirtHandlerNode, &Args);
4059	Args = s_RightToLeft;
4060	cErrors += RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesR3->PhysToVirtHandlers, false, pgmR3CheckIntegrityPhysToVirtHandlerNode, &Args);
4061
4062	return !cErrors ? VINF_SUCCESS : VERR_INTERNAL_ERROR;
4063	}
4064

Note: See TracBrowser for help on using the repository browser.

source: vbox/trunk/src/VBox/VMM/VMMR3/PGM.cpp@ 36701

Download in other formats: