PGM.cpp@ 37423

Last change on this file since 37423 was 37354, checked in by vboxsync, 14 years ago
PGM: Fixed locking issues in PGMR3PhysMMIORegister and PGMR3PhysMMIODeregister. Also addressed a harmless on in PGMR3PhysRomRegister (only used at init time, so no races). Fortified the code with assertions more lock assertion, replacing the incorrect PGMIsLocked() checks (we only care if the current thread is the lock owner). Cleaned up some ReturnStmt macros and adding more of them.
Property svn:eol-style set to `native` Property svn:keywords set to `Id`
File size: 199.8 KB

Line
1	/* $Id: PGM.cpp 37354 2011-06-07 15:05:32Z vboxsync $ */
2	/** @file
3	* PGM - Page Manager and Monitor. (Mixing stuff here, not good?)
4	*/
5
6	/*
7	* Copyright (C) 2006-2010 Oracle Corporation
8	*
9	* This file is part of VirtualBox Open Source Edition (OSE), as
10	* available from http://www.virtualbox.org. This file is free software;
11	* you can redistribute it and/or modify it under the terms of the GNU
12	* General Public License (GPL) as published by the Free Software
13	* Foundation, in version 2 as it comes in the "COPYING" file of the
14	* VirtualBox OSE distribution. VirtualBox OSE is distributed in the
15	* hope that it will be useful, but WITHOUT ANY WARRANTY of any kind.
16	*/
17
18
19	/** @page pg_pgm PGM - The Page Manager and Monitor
20	*
21	* @see grp_pgm,
22	* @ref pg_pgm_pool,
23	* @ref pg_pgm_phys.
24	*
25	*
26	* @section sec_pgm_modes Paging Modes
27	*
28	* There are three memory contexts: Host Context (HC), Guest Context (GC)
29	* and intermediate context. When talking about paging HC can also be referred to
30	* as "host paging", and GC referred to as "shadow paging".
31	*
32	* We define three basic paging modes: 32-bit, PAE and AMD64. The host paging mode
33	* is defined by the host operating system. The mode used in the shadow paging mode
34	* depends on the host paging mode and what the mode the guest is currently in. The
35	* following relation between the two is defined:
36	*
37	* @verbatim
38	Host > 32-bit \| PAE \| AMD64 \|
39	Guest \| \| \| \|
40	==v================================
41	32-bit 32-bit PAE PAE
42	-------\|--------\|--------\|--------\|
43	PAE PAE PAE PAE
44	-------\|--------\|--------\|--------\|
45	AMD64 AMD64 AMD64 AMD64
46	-------\|--------\|--------\|--------\| @endverbatim
47	*
48	* All configuration except those in the diagonal (upper left) are expected to
49	* require special effort from the switcher (i.e. a bit slower).
50	*
51	*
52	*
53	*
54	* @section sec_pgm_shw The Shadow Memory Context
55	*
56	*
57	* [..]
58	*
59	* Because of guest context mappings requires PDPT and PML4 entries to allow
60	* writing on AMD64, the two upper levels will have fixed flags whatever the
61	* guest is thinking of using there. So, when shadowing the PD level we will
62	* calculate the effective flags of PD and all the higher levels. In legacy
63	* PAE mode this only applies to the PWT and PCD bits (the rest are
64	* ignored/reserved/MBZ). We will ignore those bits for the present.
65	*
66	*
67	*
68	* @section sec_pgm_int The Intermediate Memory Context
69	*
70	* The world switch goes thru an intermediate memory context which purpose it is
71	* to provide different mappings of the switcher code. All guest mappings are also
72	* present in this context.
73	*
74	* The switcher code is mapped at the same location as on the host, at an
75	* identity mapped location (physical equals virtual address), and at the
76	* hypervisor location. The identity mapped location is for when the world
77	* switches that involves disabling paging.
78	*
79	* PGM maintain page tables for 32-bit, PAE and AMD64 paging modes. This
80	* simplifies switching guest CPU mode and consistency at the cost of more
81	* code to do the work. All memory use for those page tables is located below
82	* 4GB (this includes page tables for guest context mappings).
83	*
84	*
85	* @subsection subsec_pgm_int_gc Guest Context Mappings
86	*
87	* During assignment and relocation of a guest context mapping the intermediate
88	* memory context is used to verify the new location.
89	*
90	* Guest context mappings are currently restricted to below 4GB, for reasons
91	* of simplicity. This may change when we implement AMD64 support.
92	*
93	*
94	*
95	*
96	* @section sec_pgm_misc Misc
97	*
98	* @subsection subsec_pgm_misc_diff Differences Between Legacy PAE and Long Mode PAE
99	*
100	* The differences between legacy PAE and long mode PAE are:
101	* -# PDPE bits 1, 2, 5 and 6 are defined differently. In leagcy mode they are
102	* all marked down as must-be-zero, while in long mode 1, 2 and 5 have the
103	* usual meanings while 6 is ignored (AMD). This means that upon switching to
104	* legacy PAE mode we'll have to clear these bits and when going to long mode
105	* they must be set. This applies to both intermediate and shadow contexts,
106	* however we don't need to do it for the intermediate one since we're
107	* executing with CR0.WP at that time.
108	* -# CR3 allows a 32-byte aligned address in legacy mode, while in long mode
109	* a page aligned one is required.
110	*
111	*
112	* @section sec_pgm_handlers Access Handlers
113	*
114	* Placeholder.
115	*
116	*
117	* @subsection sec_pgm_handlers_virt Virtual Access Handlers
118	*
119	* Placeholder.
120	*
121	*
122	* @subsection sec_pgm_handlers_virt Virtual Access Handlers
123	*
124	* We currently implement three types of virtual access handlers: ALL, WRITE
125	* and HYPERVISOR (WRITE). See PGMVIRTHANDLERTYPE for some more details.
126	*
127	* The HYPERVISOR access handlers is kept in a separate tree since it doesn't apply
128	* to physical pages (PGMTREES::HyperVirtHandlers) and only needs to be consulted in
129	* a special \#PF case. The ALL and WRITE are in the PGMTREES::VirtHandlers tree, the
130	* rest of this section is going to be about these handlers.
131	*
132	* We'll go thru the life cycle of a handler and try make sense of it all, don't know
133	* how successful this is gonna be...
134	*
135	* 1. A handler is registered thru the PGMR3HandlerVirtualRegister and
136	* PGMHandlerVirtualRegisterEx APIs. We check for conflicting virtual handlers
137	* and create a new node that is inserted into the AVL tree (range key). Then
138	* a full PGM resync is flagged (clear pool, sync cr3, update virtual bit of PGMPAGE).
139	*
140	* 2. The following PGMSyncCR3/SyncCR3 operation will first make invoke HandlerVirtualUpdate.
141	*
142	* 2a. HandlerVirtualUpdate will will lookup all the pages covered by virtual handlers
143	* via the current guest CR3 and update the physical page -> virtual handler
144	* translation. Needless to say, this doesn't exactly scale very well. If any changes
145	* are detected, it will flag a virtual bit update just like we did on registration.
146	* PGMPHYS pages with changes will have their virtual handler state reset to NONE.
147	*
148	* 2b. The virtual bit update process will iterate all the pages covered by all the
149	* virtual handlers and update the PGMPAGE virtual handler state to the max of all
150	* virtual handlers on that page.
151	*
152	* 2c. Back in SyncCR3 we will now flush the entire shadow page cache to make sure
153	* we don't miss any alias mappings of the monitored pages.
154	*
155	* 2d. SyncCR3 will then proceed with syncing the CR3 table.
156	*
157	* 3. \#PF(np,read) on a page in the range. This will cause it to be synced
158	* read-only and resumed if it's a WRITE handler. If it's an ALL handler we
159	* will call the handlers like in the next step. If the physical mapping has
160	* changed we will - some time in the future - perform a handler callback
161	* (optional) and update the physical -> virtual handler cache.
162	*
163	* 4. \#PF(,write) on a page in the range. This will cause the handler to
164	* be invoked.
165	*
166	* 5. The guest invalidates the page and changes the physical backing or
167	* unmaps it. This should cause the invalidation callback to be invoked
168	* (it might not yet be 100% perfect). Exactly what happens next... is
169	* this where we mess up and end up out of sync for a while?
170	*
171	* 6. The handler is deregistered by the client via PGMHandlerVirtualDeregister.
172	* We will then set all PGMPAGEs in the physical -> virtual handler cache for
173	* this handler to NONE and trigger a full PGM resync (basically the same
174	* as int step 1). Which means 2 is executed again.
175	*
176	*
177	* @subsubsection sub_sec_pgm_handler_virt_todo TODOs
178	*
179	* There is a bunch of things that needs to be done to make the virtual handlers
180	* work 100% correctly and work more efficiently.
181	*
182	* The first bit hasn't been implemented yet because it's going to slow the
183	* whole mess down even more, and besides it seems to be working reliably for
184	* our current uses. OTOH, some of the optimizations might end up more or less
185	* implementing the missing bits, so we'll see.
186	*
187	* On the optimization side, the first thing to do is to try avoid unnecessary
188	* cache flushing. Then try team up with the shadowing code to track changes
189	* in mappings by means of access to them (shadow in), updates to shadows pages,
190	* invlpg, and shadow PT discarding (perhaps).
191	*
192	* Some idea that have popped up for optimization for current and new features:
193	* - bitmap indicating where there are virtual handlers installed.
194	* (4KB => 220 pages, page 212 => covers 32-bit address space 1:1!)
195	* - Further optimize this by min/max (needs min/max avl getters).
196	* - Shadow page table entry bit (if any left)?
197	*
198	*/
199
200
201	/** @page pg_pgm_phys PGM Physical Guest Memory Management
202	*
203	*
204	* Objectives:
205	* - Guest RAM over-commitment using memory ballooning,
206	* zero pages and general page sharing.
207	* - Moving or mirroring a VM onto a different physical machine.
208	*
209	*
210	* @subsection subsec_pgmPhys_Definitions Definitions
211	*
212	* Allocation chunk - A RTR0MemObjAllocPhysNC object and the tracking
213	* machinery associated with it.
214	*
215	*
216	*
217	*
218	* @subsection subsec_pgmPhys_AllocPage Allocating a page.
219	*
220	* Initially we map all guest memory to the (per VM) zero page, which
221	* means that none of the read functions will cause pages to be allocated.
222	*
223	* Exception, access bit in page tables that have been shared. This must
224	* be handled, but we must also make sure PGMGst*Modify doesn't make
225	* unnecessary modifications.
226	*
227	* Allocation points:
228	* - PGMPhysSimpleWriteGCPhys and PGMPhysWrite.
229	* - Replacing a zero page mapping at \#PF.
230	* - Replacing a shared page mapping at \#PF.
231	* - ROM registration (currently MMR3RomRegister).
232	* - VM restore (pgmR3Load).
233	*
234	* For the first three it would make sense to keep a few pages handy
235	* until we've reached the max memory commitment for the VM.
236	*
237	* For the ROM registration, we know exactly how many pages we need
238	* and will request these from ring-0. For restore, we will save
239	* the number of non-zero pages in the saved state and allocate
240	* them up front. This would allow the ring-0 component to refuse
241	* the request if the isn't sufficient memory available for VM use.
242	*
243	* Btw. for both ROM and restore allocations we won't be requiring
244	* zeroed pages as they are going to be filled instantly.
245	*
246	*
247	* @subsection subsec_pgmPhys_FreePage Freeing a page
248	*
249	* There are a few points where a page can be freed:
250	* - After being replaced by the zero page.
251	* - After being replaced by a shared page.
252	* - After being ballooned by the guest additions.
253	* - At reset.
254	* - At restore.
255	*
256	* When freeing one or more pages they will be returned to the ring-0
257	* component and replaced by the zero page.
258	*
259	* The reasoning for clearing out all the pages on reset is that it will
260	* return us to the exact same state as on power on, and may thereby help
261	* us reduce the memory load on the system. Further it might have a
262	* (temporary) positive influence on memory fragmentation (@see subsec_pgmPhys_Fragmentation).
263	*
264	* On restore, as mention under the allocation topic, pages should be
265	* freed / allocated depending on how many is actually required by the
266	* new VM state. The simplest approach is to do like on reset, and free
267	* all non-ROM pages and then allocate what we need.
268	*
269	* A measure to prevent some fragmentation, would be to let each allocation
270	* chunk have some affinity towards the VM having allocated the most pages
271	* from it. Also, try make sure to allocate from allocation chunks that
272	* are almost full. Admittedly, both these measures might work counter to
273	* our intentions and its probably not worth putting a lot of effort,
274	* cpu time or memory into this.
275	*
276	*
277	* @subsection subsec_pgmPhys_SharePage Sharing a page
278	*
279	* The basic idea is that there there will be a idle priority kernel
280	* thread walking the non-shared VM pages hashing them and looking for
281	* pages with the same checksum. If such pages are found, it will compare
282	* them byte-by-byte to see if they actually are identical. If found to be
283	* identical it will allocate a shared page, copy the content, check that
284	* the page didn't change while doing this, and finally request both the
285	* VMs to use the shared page instead. If the page is all zeros (special
286	* checksum and byte-by-byte check) it will request the VM that owns it
287	* to replace it with the zero page.
288	*
289	* To make this efficient, we will have to make sure not to try share a page
290	* that will change its contents soon. This part requires the most work.
291	* A simple idea would be to request the VM to write monitor the page for
292	* a while to make sure it isn't modified any time soon. Also, it may
293	* make sense to skip pages that are being write monitored since this
294	* information is readily available to the thread if it works on the
295	* per-VM guest memory structures (presently called PGMRAMRANGE).
296	*
297	*
298	* @subsection subsec_pgmPhys_Fragmentation Fragmentation Concerns and Counter Measures
299	*
300	* The pages are organized in allocation chunks in ring-0, this is a necessity
301	* if we wish to have an OS agnostic approach to this whole thing. (On Linux we
302	* could easily work on a page-by-page basis if we liked. Whether this is possible
303	* or efficient on NT I don't quite know.) Fragmentation within these chunks may
304	* become a problem as part of the idea here is that we wish to return memory to
305	* the host system.
306	*
307	* For instance, starting two VMs at the same time, they will both allocate the
308	* guest memory on-demand and if permitted their page allocations will be
309	* intermixed. Shut down one of the two VMs and it will be difficult to return
310	* any memory to the host system because the page allocation for the two VMs are
311	* mixed up in the same allocation chunks.
312	*
313	* To further complicate matters, when pages are freed because they have been
314	* ballooned or become shared/zero the whole idea is that the page is supposed
315	* to be reused by another VM or returned to the host system. This will cause
316	* allocation chunks to contain pages belonging to different VMs and prevent
317	* returning memory to the host when one of those VM shuts down.
318	*
319	* The only way to really deal with this problem is to move pages. This can
320	* either be done at VM shutdown and or by the idle priority worker thread
321	* that will be responsible for finding sharable/zero pages. The mechanisms
322	* involved for coercing a VM to move a page (or to do it for it) will be
323	* the same as when telling it to share/zero a page.
324	*
325	*
326	* @subsection subsec_pgmPhys_Tracking Tracking Structures And Their Cost
327	*
328	* There's a difficult balance between keeping the per-page tracking structures
329	* (global and guest page) easy to use and keeping them from eating too much
330	* memory. We have limited virtual memory resources available when operating in
331	* 32-bit kernel space (on 64-bit there'll it's quite a different story). The
332	* tracking structures will be attempted designed such that we can deal with up
333	* to 32GB of memory on a 32-bit system and essentially unlimited on 64-bit ones.
334	*
335	*
336	* @subsubsection subsubsec_pgmPhys_Tracking_Kernel Kernel Space
337	*
338	* @see pg_GMM
339	*
340	* @subsubsection subsubsec_pgmPhys_Tracking_PerVM Per-VM
341	*
342	* Fixed info is the physical address of the page (HCPhys) and the page id
343	* (described above). Theoretically we'll need 48(-12) bits for the HCPhys part.
344	* Today we've restricting ourselves to 40(-12) bits because this is the current
345	* restrictions of all AMD64 implementations (I think Barcelona will up this
346	* to 48(-12) bits, not that it really matters) and I needed the bits for
347	* tracking mappings of a page. 48-12 = 36. That leaves 28 bits, which means a
348	* decent range for the page id: 2^(28+12) = 1024TB.
349	*
350	* In additions to these, we'll have to keep maintaining the page flags as we
351	* currently do. Although it wouldn't harm to optimize these quite a bit, like
352	* for instance the ROM shouldn't depend on having a write handler installed
353	* in order for it to become read-only. A RO/RW bit should be considered so
354	* that the page syncing code doesn't have to mess about checking multiple
355	* flag combinations (ROM \|\| RW handler \|\| write monitored) in order to
356	* figure out how to setup a shadow PTE. But this of course, is second
357	* priority at present. Current this requires 12 bits, but could probably
358	* be optimized to ~8.
359	*
360	* Then there's the 24 bits used to track which shadow page tables are
361	* currently mapping a page for the purpose of speeding up physical
362	* access handlers, and thereby the page pool cache. More bit for this
363	* purpose wouldn't hurt IIRC.
364	*
365	* Then there is a new bit in which we need to record what kind of page
366	* this is, shared, zero, normal or write-monitored-normal. This'll
367	* require 2 bits. One bit might be needed for indicating whether a
368	* write monitored page has been written to. And yet another one or
369	* two for tracking migration status. 3-4 bits total then.
370	*
371	* Whatever is left will can be used to record the sharabilitiy of a
372	* page. The page checksum will not be stored in the per-VM table as
373	* the idle thread will not be permitted to do modifications to it.
374	* It will instead have to keep its own working set of potentially
375	* shareable pages and their check sums and stuff.
376	*
377	* For the present we'll keep the current packing of the
378	* PGMRAMRANGE::aHCPhys to keep the changes simple, only of course,
379	* we'll have to change it to a struct with a total of 128-bits at
380	* our disposal.
381	*
382	* The initial layout will be like this:
383	* @verbatim
384	RTHCPHYS HCPhys; The current stuff.
385	63:40 Current shadow PT tracking stuff.
386	39:12 The physical page frame number.
387	11:0 The current flags.
388	uint32_t u28PageId : 28; The page id.
389	uint32_t u2State : 2; The page state { zero, shared, normal, write monitored }.
390	uint32_t fWrittenTo : 1; Whether a write monitored page was written to.
391	uint32_t u1Reserved : 1; Reserved for later.
392	uint32_t u32Reserved; Reserved for later, mostly sharing stats.
393	@endverbatim
394	*
395	* The final layout will be something like this:
396	* @verbatim
397	RTHCPHYS HCPhys; The current stuff.
398	63:48 High page id (12+).
399	47:12 The physical page frame number.
400	11:0 Low page id.
401	uint32_t fReadOnly : 1; Whether it's readonly page (rom or monitored in some way).
402	uint32_t u3Type : 3; The page type {RESERVED, MMIO, MMIO2, ROM, shadowed ROM, RAM}.
403	uint32_t u2PhysMon : 2; Physical access handler type {none, read, write, all}.
404	uint32_t u2VirtMon : 2; Virtual access handler type {none, read, write, all}..
405	uint32_t u2State : 2; The page state { zero, shared, normal, write monitored }.
406	uint32_t fWrittenTo : 1; Whether a write monitored page was written to.
407	uint32_t u20Reserved : 20; Reserved for later, mostly sharing stats.
408	uint32_t u32Tracking; The shadow PT tracking stuff, roughly.
409	@endverbatim
410	*
411	* Cost wise, this means we'll double the cost for guest memory. There isn't anyway
412	* around that I'm afraid. It means that the cost of dealing out 32GB of memory
413	* to one or more VMs is: (32GB >> PAGE_SHIFT) * 16 bytes, or 128MBs. Or another
414	* example, the VM heap cost when assigning 1GB to a VM will be: 4MB.
415	*
416	* A couple of cost examples for the total cost per-VM + kernel.
417	* 32-bit Windows and 32-bit linux:
418	* 1GB guest ram, 256K pages: 4MB + 2MB(+) = 6MB
419	* 4GB guest ram, 1M pages: 16MB + 8MB(+) = 24MB
420	* 32GB guest ram, 8M pages: 128MB + 64MB(+) = 192MB
421	* 64-bit Windows and 64-bit linux:
422	* 1GB guest ram, 256K pages: 4MB + 3MB(+) = 7MB
423	* 4GB guest ram, 1M pages: 16MB + 12MB(+) = 28MB
424	* 32GB guest ram, 8M pages: 128MB + 96MB(+) = 224MB
425	*
426	* UPDATE - 2007-09-27:
427	* Will need a ballooned flag/state too because we cannot
428	* trust the guest 100% and reporting the same page as ballooned more
429	* than once will put the GMM off balance.
430	*
431	*
432	* @subsection subsec_pgmPhys_Serializing Serializing Access
433	*
434	* Initially, we'll try a simple scheme:
435	*
436	* - The per-VM RAM tracking structures (PGMRAMRANGE) is only modified
437	* by the EMT thread of that VM while in the pgm critsect.
438	* - Other threads in the VM process that needs to make reliable use of
439	* the per-VM RAM tracking structures will enter the critsect.
440	* - No process external thread or kernel thread will ever try enter
441	* the pgm critical section, as that just won't work.
442	* - The idle thread (and similar threads) doesn't not need 100% reliable
443	* data when performing it tasks as the EMT thread will be the one to
444	* do the actual changes later anyway. So, as long as it only accesses
445	* the main ram range, it can do so by somehow preventing the VM from
446	* being destroyed while it works on it...
447	*
448	* - The over-commitment management, including the allocating/freeing
449	* chunks, is serialized by a ring-0 mutex lock (a fast one since the
450	* more mundane mutex implementation is broken on Linux).
451	* - A separate mutex is protecting the set of allocation chunks so
452	* that pages can be shared or/and freed up while some other VM is
453	* allocating more chunks. This mutex can be take from under the other
454	* one, but not the other way around.
455	*
456	*
457	* @subsection subsec_pgmPhys_Request VM Request interface
458	*
459	* When in ring-0 it will become necessary to send requests to a VM so it can
460	* for instance move a page while defragmenting during VM destroy. The idle
461	* thread will make use of this interface to request VMs to setup shared
462	* pages and to perform write monitoring of pages.
463	*
464	* I would propose an interface similar to the current VMReq interface, similar
465	* in that it doesn't require locking and that the one sending the request may
466	* wait for completion if it wishes to. This shouldn't be very difficult to
467	* realize.
468	*
469	* The requests themselves are also pretty simple. They are basically:
470	* -# Check that some precondition is still true.
471	* -# Do the update.
472	* -# Update all shadow page tables involved with the page.
473	*
474	* The 3rd step is identical to what we're already doing when updating a
475	* physical handler, see pgmHandlerPhysicalSetRamFlagsAndFlushShadowPTs.
476	*
477	*
478	*
479	* @section sec_pgmPhys_MappingCaches Mapping Caches
480	*
481	* In order to be able to map in and out memory and to be able to support
482	* guest with more RAM than we've got virtual address space, we'll employing
483	* a mapping cache. Normally ring-0 and ring-3 can share the same cache,
484	* however on 32-bit darwin the ring-0 code is running in a different memory
485	* context and therefore needs a separate cache. In raw-mode context we also
486	* need a separate cache. The 32-bit darwin mapping cache and the one for
487	* raw-mode context share a lot of code, see PGMRZDYNMAP.
488	*
489	*
490	* @subsection subsec_pgmPhys_MappingCaches_R3 Ring-3
491	*
492	* We've considered implementing the ring-3 mapping cache page based but found
493	* that this was bother some when one had to take into account TLBs+SMP and
494	* portability (missing the necessary APIs on several platforms). There were
495	* also some performance concerns with this approach which hadn't quite been
496	* worked out.
497	*
498	* Instead, we'll be mapping allocation chunks into the VM process. This simplifies
499	* matters greatly quite a bit since we don't need to invent any new ring-0 stuff,
500	* only some minor RTR0MEMOBJ mapping stuff. The main concern here is that mapping
501	* compared to the previous idea is that mapping or unmapping a 1MB chunk is more
502	* costly than a single page, although how much more costly is uncertain. We'll
503	* try address this by using a very big cache, preferably bigger than the actual
504	* VM RAM size if possible. The current VM RAM sizes should give some idea for
505	* 32-bit boxes, while on 64-bit we can probably get away with employing an
506	* unlimited cache.
507	*
508	* The cache have to parts, as already indicated, the ring-3 side and the
509	* ring-0 side.
510	*
511	* The ring-0 will be tied to the page allocator since it will operate on the
512	* memory objects it contains. It will therefore require the first ring-0 mutex
513	* discussed in @ref subsec_pgmPhys_Serializing. We
514	* some double house keeping wrt to who has mapped what I think, since both
515	* VMMR0.r0 and RTR0MemObj will keep track of mapping relations
516	*
517	* The ring-3 part will be protected by the pgm critsect. For simplicity, we'll
518	* require anyone that desires to do changes to the mapping cache to do that
519	* from within this critsect. Alternatively, we could employ a separate critsect
520	* for serializing changes to the mapping cache as this would reduce potential
521	* contention with other threads accessing mappings unrelated to the changes
522	* that are in process. We can see about this later, contention will show
523	* up in the statistics anyway, so it'll be simple to tell.
524	*
525	* The organization of the ring-3 part will be very much like how the allocation
526	* chunks are organized in ring-0, that is in an AVL tree by chunk id. To avoid
527	* having to walk the tree all the time, we'll have a couple of lookaside entries
528	* like in we do for I/O ports and MMIO in IOM.
529	*
530	* The simplified flow of a PGMPhysRead/Write function:
531	* -# Enter the PGM critsect.
532	* -# Lookup GCPhys in the ram ranges and get the Page ID.
533	* -# Calc the Allocation Chunk ID from the Page ID.
534	* -# Check the lookaside entries and then the AVL tree for the Chunk ID.
535	* If not found in cache:
536	* -# Call ring-0 and request it to be mapped and supply
537	* a chunk to be unmapped if the cache is maxed out already.
538	* -# Insert the new mapping into the AVL tree (id + R3 address).
539	* -# Update the relevant lookaside entry and return the mapping address.
540	* -# Do the read/write according to monitoring flags and everything.
541	* -# Leave the critsect.
542	*
543	*
544	* @section sec_pgmPhys_Fallback Fallback
545	*
546	* Current all the "second tier" hosts will not support the RTR0MemObjAllocPhysNC
547	* API and thus require a fallback.
548	*
549	* So, when RTR0MemObjAllocPhysNC returns VERR_NOT_SUPPORTED the page allocator
550	* will return to the ring-3 caller (and later ring-0) and asking it to seed
551	* the page allocator with some fresh pages (VERR_GMM_SEED_ME). Ring-3 will
552	* then perform an SUPR3PageAlloc(cbChunk >> PAGE_SHIFT) call and make a
553	* "SeededAllocPages" call to ring-0.
554	*
555	* The first time ring-0 sees the VERR_NOT_SUPPORTED failure it will disable
556	* all page sharing (zero page detection will continue). It will also force
557	* all allocations to come from the VM which seeded the page. Both these
558	* measures are taken to make sure that there will never be any need for
559	* mapping anything into ring-3 - everything will be mapped already.
560	*
561	* Whether we'll continue to use the current MM locked memory management
562	* for this I don't quite know (I'd prefer not to and just ditch that all
563	* together), we'll see what's simplest to do.
564	*
565	*
566	*
567	* @section sec_pgmPhys_Changes Changes
568	*
569	* Breakdown of the changes involved?
570	*/
571
572	/*******************************************************************************
573	* Header Files *
574	*******************************************************************************/
575	#define LOG_GROUP LOG_GROUP_PGM
576	#include <VBox/vmm/dbgf.h>
577	#include <VBox/vmm/pgm.h>
578	#include <VBox/vmm/cpum.h>
579	#include <VBox/vmm/iom.h>
580	#include <VBox/sup.h>
581	#include <VBox/vmm/mm.h>
582	#include <VBox/vmm/em.h>
583	#include <VBox/vmm/stam.h>
584	#include <VBox/vmm/rem.h>
585	#include <VBox/vmm/selm.h>
586	#include <VBox/vmm/ssm.h>
587	#include <VBox/vmm/hwaccm.h>
588	#include "PGMInternal.h"
589	#include <VBox/vmm/vm.h>
590	#include "PGMInline.h"
591
592	#include <VBox/dbg.h>
593	#include <VBox/param.h>
594	#include <VBox/err.h>
595
596	#include <iprt/asm.h>
597	#include <iprt/asm-amd64-x86.h>
598	#include <iprt/assert.h>
599	#include <iprt/env.h>
600	#include <iprt/mem.h>
601	#include <iprt/file.h>
602	#include <iprt/string.h>
603	#include <iprt/thread.h>
604
605
606	/*******************************************************************************
607	* Internal Functions *
608	*******************************************************************************/
609	static int pgmR3InitPaging(PVM pVM);
610	static int pgmR3InitStats(PVM pVM);
611	static DECLCALLBACK(void) pgmR3PhysInfo(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs);
612	static DECLCALLBACK(void) pgmR3InfoMode(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs);
613	static DECLCALLBACK(void) pgmR3InfoCr3(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs);
614	static DECLCALLBACK(int) pgmR3RelocatePhysHandler(PAVLROGCPHYSNODECORE pNode, void *pvUser);
615	static DECLCALLBACK(int) pgmR3RelocateVirtHandler(PAVLROGCPTRNODECORE pNode, void *pvUser);
616	static DECLCALLBACK(int) pgmR3RelocateHyperVirtHandler(PAVLROGCPTRNODECORE pNode, void *pvUser);
617	#ifdef VBOX_STRICT
618	static DECLCALLBACK(void) pgmR3ResetNoMorePhysWritesFlag(PVM pVM, VMSTATE enmState, VMSTATE enmOldState, void *pvUser);
619	#endif
620	static int pgmR3ModeDataInit(PVM pVM, bool fResolveGCAndR0);
621	static void pgmR3ModeDataSwitch(PVM pVM, PVMCPU pVCpu, PGMMODE enmShw, PGMMODE enmGst);
622	static PGMMODE pgmR3CalcShadowMode(PVM pVM, PGMMODE enmGuestMode, SUPPAGINGMODE enmHostMode, PGMMODE enmShadowMode, VMMSWITCHER *penmSwitcher);
623
624	#ifdef VBOX_WITH_DEBUGGER
625	/** @todo Convert the first two commands to 'info' items. */
626	static DECLCALLBACK(int) pgmR3CmdRam(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs);
627	static DECLCALLBACK(int) pgmR3CmdError(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs);
628	static DECLCALLBACK(int) pgmR3CmdSync(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs);
629	static DECLCALLBACK(int) pgmR3CmdSyncAlways(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs);
630	# ifdef VBOX_STRICT
631	static DECLCALLBACK(int) pgmR3CmdAssertCR3(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs);
632	# endif
633	static DECLCALLBACK(int) pgmR3CmdPhysToFile(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs);
634	#endif
635
636
637	/*******************************************************************************
638	* Global Variables *
639	*******************************************************************************/
640	#ifdef VBOX_WITH_DEBUGGER
641	/** Argument descriptors for '.pgmerror' and '.pgmerroroff'. */
642	static const DBGCVARDESC g_aPgmErrorArgs[] =
643	{
644	/* cTimesMin, cTimesMax, enmCategory, fFlags, pszName, pszDescription */
645	{ 0, 1, DBGCVAR_CAT_STRING, 0, "where", "Error injection location." },
646	};
647
648	static const DBGCVARDESC g_aPgmPhysToFileArgs[] =
649	{
650	/* cTimesMin, cTimesMax, enmCategory, fFlags, pszName, pszDescription */
651	{ 1, 1, DBGCVAR_CAT_STRING, 0, "file", "The file name." },
652	{ 0, 1, DBGCVAR_CAT_STRING, 0, "nozero", "If present, zero pages are skipped." },
653	};
654
655	# ifdef DEBUG_sandervl
656	static const DBGCVARDESC g_aPgmCountPhysWritesArgs[] =
657	{
658	/* cTimesMin, cTimesMax, enmCategory, fFlags, pszName, pszDescription */
659	{ 1, 1, DBGCVAR_CAT_STRING, 0, "enabled", "on/off." },
660	{ 1, 1, DBGCVAR_CAT_NUMBER_NO_RANGE, 0, "interval", "Interval in ms." },
661	};
662	# endif
663
664	/** Command descriptors. */
665	static const DBGCCMD g_aCmds[] =
666	{
667	/* pszCmd, cArgsMin, cArgsMax, paArgDesc, cArgDescs, fFlags, pfnHandler pszSyntax, ....pszDescription */
668	{ "pgmram", 0, 0, NULL, 0, 0, pgmR3CmdRam, "", "Display the ram ranges." },
669	{ "pgmsync", 0, 0, NULL, 0, 0, pgmR3CmdSync, "", "Sync the CR3 page." },
670	{ "pgmerror", 0, 1, &g_aPgmErrorArgs[0], 1, 0, pgmR3CmdError, "", "Enables inject runtime of errors into parts of PGM." },
671	{ "pgmerroroff", 0, 1, &g_aPgmErrorArgs[0], 1, 0, pgmR3CmdError, "", "Disables inject runtime errors into parts of PGM." },
672	# ifdef VBOX_STRICT
673	{ "pgmassertcr3", 0, 0, NULL, 0, 0, pgmR3CmdAssertCR3, "", "Check the shadow CR3 mapping." },
674	# if HC_ARCH_BITS == 64
675	{ "pgmcheckduppages", 0, 0, NULL, 0, 0, pgmR3CmdCheckDuplicatePages, "", "Check for duplicate pages in all running VMs." },
676	{ "pgmsharedmodules", 0, 0, NULL, 0, 0, pgmR3CmdShowSharedModules, "", "Print shared modules info." },
677	# endif
678	# endif
679	{ "pgmsyncalways", 0, 0, NULL, 0, 0, pgmR3CmdSyncAlways, "", "Toggle permanent CR3 syncing." },
680	{ "pgmphystofile", 1, 2, &g_aPgmPhysToFileArgs[0], 2, 0, pgmR3CmdPhysToFile, "", "Save the physical memory to file." },
681	};
682	#endif
683
684
685
686
687	/*
688	* Shadow - 32-bit mode
689	*/
690	#define PGM_SHW_TYPE PGM_TYPE_32BIT
691	#define PGM_SHW_NAME(name) PGM_SHW_NAME_32BIT(name)
692	#define PGM_SHW_NAME_RC_STR(name) PGM_SHW_NAME_RC_32BIT_STR(name)
693	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_32BIT_STR(name)
694	#include "PGMShw.h"
695
696	/* Guest - real mode */
697	#define PGM_GST_TYPE PGM_TYPE_REAL
698	#define PGM_GST_NAME(name) PGM_GST_NAME_REAL(name)
699	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_REAL_STR(name)
700	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_REAL_STR(name)
701	#define PGM_BTH_NAME(name) PGM_BTH_NAME_32BIT_REAL(name)
702	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_32BIT_REAL_STR(name)
703	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_32BIT_REAL_STR(name)
704	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_32BIT_PT_FOR_PHYS
705	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_32BIT_PD_PHYS
706	#include "PGMBth.h"
707	#include "PGMGstDefs.h"
708	#include "PGMGst.h"
709	#undef BTH_PGMPOOLKIND_PT_FOR_PT
710	#undef BTH_PGMPOOLKIND_ROOT
711	#undef PGM_BTH_NAME
712	#undef PGM_BTH_NAME_RC_STR
713	#undef PGM_BTH_NAME_R0_STR
714	#undef PGM_GST_TYPE
715	#undef PGM_GST_NAME
716	#undef PGM_GST_NAME_RC_STR
717	#undef PGM_GST_NAME_R0_STR
718
719	/* Guest - protected mode */
720	#define PGM_GST_TYPE PGM_TYPE_PROT
721	#define PGM_GST_NAME(name) PGM_GST_NAME_PROT(name)
722	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PROT_STR(name)
723	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PROT_STR(name)
724	#define PGM_BTH_NAME(name) PGM_BTH_NAME_32BIT_PROT(name)
725	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_32BIT_PROT_STR(name)
726	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_32BIT_PROT_STR(name)
727	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_32BIT_PT_FOR_PHYS
728	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_32BIT_PD_PHYS
729	#include "PGMBth.h"
730	#include "PGMGstDefs.h"
731	#include "PGMGst.h"
732	#undef BTH_PGMPOOLKIND_PT_FOR_PT
733	#undef BTH_PGMPOOLKIND_ROOT
734	#undef PGM_BTH_NAME
735	#undef PGM_BTH_NAME_RC_STR
736	#undef PGM_BTH_NAME_R0_STR
737	#undef PGM_GST_TYPE
738	#undef PGM_GST_NAME
739	#undef PGM_GST_NAME_RC_STR
740	#undef PGM_GST_NAME_R0_STR
741
742	/* Guest - 32-bit mode */
743	#define PGM_GST_TYPE PGM_TYPE_32BIT
744	#define PGM_GST_NAME(name) PGM_GST_NAME_32BIT(name)
745	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_32BIT_STR(name)
746	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_32BIT_STR(name)
747	#define PGM_BTH_NAME(name) PGM_BTH_NAME_32BIT_32BIT(name)
748	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_32BIT_32BIT_STR(name)
749	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_32BIT_32BIT_STR(name)
750	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_32BIT_PT_FOR_32BIT_PT
751	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_32BIT_PT_FOR_32BIT_4MB
752	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_32BIT_PD
753	#include "PGMBth.h"
754	#include "PGMGstDefs.h"
755	#include "PGMGst.h"
756	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
757	#undef BTH_PGMPOOLKIND_PT_FOR_PT
758	#undef BTH_PGMPOOLKIND_ROOT
759	#undef PGM_BTH_NAME
760	#undef PGM_BTH_NAME_RC_STR
761	#undef PGM_BTH_NAME_R0_STR
762	#undef PGM_GST_TYPE
763	#undef PGM_GST_NAME
764	#undef PGM_GST_NAME_RC_STR
765	#undef PGM_GST_NAME_R0_STR
766
767	#undef PGM_SHW_TYPE
768	#undef PGM_SHW_NAME
769	#undef PGM_SHW_NAME_RC_STR
770	#undef PGM_SHW_NAME_R0_STR
771
772
773	/*
774	* Shadow - PAE mode
775	*/
776	#define PGM_SHW_TYPE PGM_TYPE_PAE
777	#define PGM_SHW_NAME(name) PGM_SHW_NAME_PAE(name)
778	#define PGM_SHW_NAME_RC_STR(name) PGM_SHW_NAME_RC_PAE_STR(name)
779	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_PAE_STR(name)
780	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_REAL(name)
781	#include "PGMShw.h"
782
783	/* Guest - real mode */
784	#define PGM_GST_TYPE PGM_TYPE_REAL
785	#define PGM_GST_NAME(name) PGM_GST_NAME_REAL(name)
786	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_REAL_STR(name)
787	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_REAL_STR(name)
788	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_REAL(name)
789	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_PAE_REAL_STR(name)
790	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_PAE_REAL_STR(name)
791	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
792	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_PAE_PDPT_PHYS
793	#include "PGMGstDefs.h"
794	#include "PGMBth.h"
795	#undef BTH_PGMPOOLKIND_PT_FOR_PT
796	#undef BTH_PGMPOOLKIND_ROOT
797	#undef PGM_BTH_NAME
798	#undef PGM_BTH_NAME_RC_STR
799	#undef PGM_BTH_NAME_R0_STR
800	#undef PGM_GST_TYPE
801	#undef PGM_GST_NAME
802	#undef PGM_GST_NAME_RC_STR
803	#undef PGM_GST_NAME_R0_STR
804
805	/* Guest - protected mode */
806	#define PGM_GST_TYPE PGM_TYPE_PROT
807	#define PGM_GST_NAME(name) PGM_GST_NAME_PROT(name)
808	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PROT_STR(name)
809	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PROT_STR(name)
810	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_PROT(name)
811	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_PAE_PROT_STR(name)
812	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_PAE_PROT_STR(name)
813	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
814	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_PAE_PDPT_PHYS
815	#include "PGMGstDefs.h"
816	#include "PGMBth.h"
817	#undef BTH_PGMPOOLKIND_PT_FOR_PT
818	#undef BTH_PGMPOOLKIND_ROOT
819	#undef PGM_BTH_NAME
820	#undef PGM_BTH_NAME_RC_STR
821	#undef PGM_BTH_NAME_R0_STR
822	#undef PGM_GST_TYPE
823	#undef PGM_GST_NAME
824	#undef PGM_GST_NAME_RC_STR
825	#undef PGM_GST_NAME_R0_STR
826
827	/* Guest - 32-bit mode */
828	#define PGM_GST_TYPE PGM_TYPE_32BIT
829	#define PGM_GST_NAME(name) PGM_GST_NAME_32BIT(name)
830	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_32BIT_STR(name)
831	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_32BIT_STR(name)
832	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_32BIT(name)
833	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_PAE_32BIT_STR(name)
834	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_PAE_32BIT_STR(name)
835	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_32BIT_PT
836	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_32BIT_4MB
837	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_PAE_PDPT_FOR_32BIT
838	#include "PGMGstDefs.h"
839	#include "PGMBth.h"
840	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
841	#undef BTH_PGMPOOLKIND_PT_FOR_PT
842	#undef BTH_PGMPOOLKIND_ROOT
843	#undef PGM_BTH_NAME
844	#undef PGM_BTH_NAME_RC_STR
845	#undef PGM_BTH_NAME_R0_STR
846	#undef PGM_GST_TYPE
847	#undef PGM_GST_NAME
848	#undef PGM_GST_NAME_RC_STR
849	#undef PGM_GST_NAME_R0_STR
850
851	/* Guest - PAE mode */
852	#define PGM_GST_TYPE PGM_TYPE_PAE
853	#define PGM_GST_NAME(name) PGM_GST_NAME_PAE(name)
854	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PAE_STR(name)
855	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PAE_STR(name)
856	#define PGM_BTH_NAME(name) PGM_BTH_NAME_PAE_PAE(name)
857	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_PAE_PAE_STR(name)
858	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_PAE_PAE_STR(name)
859	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
860	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
861	#define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_PAE_PDPT
862	#include "PGMBth.h"
863	#include "PGMGstDefs.h"
864	#include "PGMGst.h"
865	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
866	#undef BTH_PGMPOOLKIND_PT_FOR_PT
867	#undef BTH_PGMPOOLKIND_ROOT
868	#undef PGM_BTH_NAME
869	#undef PGM_BTH_NAME_RC_STR
870	#undef PGM_BTH_NAME_R0_STR
871	#undef PGM_GST_TYPE
872	#undef PGM_GST_NAME
873	#undef PGM_GST_NAME_RC_STR
874	#undef PGM_GST_NAME_R0_STR
875
876	#undef PGM_SHW_TYPE
877	#undef PGM_SHW_NAME
878	#undef PGM_SHW_NAME_RC_STR
879	#undef PGM_SHW_NAME_R0_STR
880
881
882	/*
883	* Shadow - AMD64 mode
884	*/
885	#define PGM_SHW_TYPE PGM_TYPE_AMD64
886	#define PGM_SHW_NAME(name) PGM_SHW_NAME_AMD64(name)
887	#define PGM_SHW_NAME_RC_STR(name) PGM_SHW_NAME_RC_AMD64_STR(name)
888	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_AMD64_STR(name)
889	#include "PGMShw.h"
890
891	#ifdef VBOX_WITH_64_BITS_GUESTS
892	/* Guest - AMD64 mode */
893	# define PGM_GST_TYPE PGM_TYPE_AMD64
894	# define PGM_GST_NAME(name) PGM_GST_NAME_AMD64(name)
895	# define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_AMD64_STR(name)
896	# define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_AMD64_STR(name)
897	# define PGM_BTH_NAME(name) PGM_BTH_NAME_AMD64_AMD64(name)
898	# define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_AMD64_AMD64_STR(name)
899	# define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_AMD64_AMD64_STR(name)
900	# define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
901	# define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
902	# define BTH_PGMPOOLKIND_ROOT PGMPOOLKIND_64BIT_PML4
903	# include "PGMBth.h"
904	# include "PGMGstDefs.h"
905	# include "PGMGst.h"
906	# undef BTH_PGMPOOLKIND_PT_FOR_BIG
907	# undef BTH_PGMPOOLKIND_PT_FOR_PT
908	# undef BTH_PGMPOOLKIND_ROOT
909	# undef PGM_BTH_NAME
910	# undef PGM_BTH_NAME_RC_STR
911	# undef PGM_BTH_NAME_R0_STR
912	# undef PGM_GST_TYPE
913	# undef PGM_GST_NAME
914	# undef PGM_GST_NAME_RC_STR
915	# undef PGM_GST_NAME_R0_STR
916	#endif /* VBOX_WITH_64_BITS_GUESTS */
917
918	#undef PGM_SHW_TYPE
919	#undef PGM_SHW_NAME
920	#undef PGM_SHW_NAME_RC_STR
921	#undef PGM_SHW_NAME_R0_STR
922
923
924	/*
925	* Shadow - Nested paging mode
926	*/
927	#define PGM_SHW_TYPE PGM_TYPE_NESTED
928	#define PGM_SHW_NAME(name) PGM_SHW_NAME_NESTED(name)
929	#define PGM_SHW_NAME_RC_STR(name) PGM_SHW_NAME_RC_NESTED_STR(name)
930	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_NESTED_STR(name)
931	#include "PGMShw.h"
932
933	/* Guest - real mode */
934	#define PGM_GST_TYPE PGM_TYPE_REAL
935	#define PGM_GST_NAME(name) PGM_GST_NAME_REAL(name)
936	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_REAL_STR(name)
937	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_REAL_STR(name)
938	#define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_REAL(name)
939	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_NESTED_REAL_STR(name)
940	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_REAL_STR(name)
941	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
942	#include "PGMGstDefs.h"
943	#include "PGMBth.h"
944	#undef BTH_PGMPOOLKIND_PT_FOR_PT
945	#undef PGM_BTH_NAME
946	#undef PGM_BTH_NAME_RC_STR
947	#undef PGM_BTH_NAME_R0_STR
948	#undef PGM_GST_TYPE
949	#undef PGM_GST_NAME
950	#undef PGM_GST_NAME_RC_STR
951	#undef PGM_GST_NAME_R0_STR
952
953	/* Guest - protected mode */
954	#define PGM_GST_TYPE PGM_TYPE_PROT
955	#define PGM_GST_NAME(name) PGM_GST_NAME_PROT(name)
956	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PROT_STR(name)
957	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PROT_STR(name)
958	#define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_PROT(name)
959	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_NESTED_PROT_STR(name)
960	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_PROT_STR(name)
961	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
962	#include "PGMGstDefs.h"
963	#include "PGMBth.h"
964	#undef BTH_PGMPOOLKIND_PT_FOR_PT
965	#undef PGM_BTH_NAME
966	#undef PGM_BTH_NAME_RC_STR
967	#undef PGM_BTH_NAME_R0_STR
968	#undef PGM_GST_TYPE
969	#undef PGM_GST_NAME
970	#undef PGM_GST_NAME_RC_STR
971	#undef PGM_GST_NAME_R0_STR
972
973	/* Guest - 32-bit mode */
974	#define PGM_GST_TYPE PGM_TYPE_32BIT
975	#define PGM_GST_NAME(name) PGM_GST_NAME_32BIT(name)
976	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_32BIT_STR(name)
977	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_32BIT_STR(name)
978	#define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_32BIT(name)
979	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_NESTED_32BIT_STR(name)
980	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_32BIT_STR(name)
981	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_32BIT_PT
982	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_32BIT_4MB
983	#include "PGMGstDefs.h"
984	#include "PGMBth.h"
985	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
986	#undef BTH_PGMPOOLKIND_PT_FOR_PT
987	#undef PGM_BTH_NAME
988	#undef PGM_BTH_NAME_RC_STR
989	#undef PGM_BTH_NAME_R0_STR
990	#undef PGM_GST_TYPE
991	#undef PGM_GST_NAME
992	#undef PGM_GST_NAME_RC_STR
993	#undef PGM_GST_NAME_R0_STR
994
995	/* Guest - PAE mode */
996	#define PGM_GST_TYPE PGM_TYPE_PAE
997	#define PGM_GST_NAME(name) PGM_GST_NAME_PAE(name)
998	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PAE_STR(name)
999	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PAE_STR(name)
1000	#define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_PAE(name)
1001	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_NESTED_PAE_STR(name)
1002	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_PAE_STR(name)
1003	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
1004	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
1005	#include "PGMGstDefs.h"
1006	#include "PGMBth.h"
1007	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
1008	#undef BTH_PGMPOOLKIND_PT_FOR_PT
1009	#undef PGM_BTH_NAME
1010	#undef PGM_BTH_NAME_RC_STR
1011	#undef PGM_BTH_NAME_R0_STR
1012	#undef PGM_GST_TYPE
1013	#undef PGM_GST_NAME
1014	#undef PGM_GST_NAME_RC_STR
1015	#undef PGM_GST_NAME_R0_STR
1016
1017	#ifdef VBOX_WITH_64_BITS_GUESTS
1018	/* Guest - AMD64 mode */
1019	# define PGM_GST_TYPE PGM_TYPE_AMD64
1020	# define PGM_GST_NAME(name) PGM_GST_NAME_AMD64(name)
1021	# define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_AMD64_STR(name)
1022	# define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_AMD64_STR(name)
1023	# define PGM_BTH_NAME(name) PGM_BTH_NAME_NESTED_AMD64(name)
1024	# define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_NESTED_AMD64_STR(name)
1025	# define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_NESTED_AMD64_STR(name)
1026	# define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
1027	# define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
1028	# include "PGMGstDefs.h"
1029	# include "PGMBth.h"
1030	# undef BTH_PGMPOOLKIND_PT_FOR_BIG
1031	# undef BTH_PGMPOOLKIND_PT_FOR_PT
1032	# undef PGM_BTH_NAME
1033	# undef PGM_BTH_NAME_RC_STR
1034	# undef PGM_BTH_NAME_R0_STR
1035	# undef PGM_GST_TYPE
1036	# undef PGM_GST_NAME
1037	# undef PGM_GST_NAME_RC_STR
1038	# undef PGM_GST_NAME_R0_STR
1039	#endif /* VBOX_WITH_64_BITS_GUESTS */
1040
1041	#undef PGM_SHW_TYPE
1042	#undef PGM_SHW_NAME
1043	#undef PGM_SHW_NAME_RC_STR
1044	#undef PGM_SHW_NAME_R0_STR
1045
1046
1047	/*
1048	* Shadow - EPT
1049	*/
1050	#define PGM_SHW_TYPE PGM_TYPE_EPT
1051	#define PGM_SHW_NAME(name) PGM_SHW_NAME_EPT(name)
1052	#define PGM_SHW_NAME_RC_STR(name) PGM_SHW_NAME_RC_EPT_STR(name)
1053	#define PGM_SHW_NAME_R0_STR(name) PGM_SHW_NAME_R0_EPT_STR(name)
1054	#include "PGMShw.h"
1055
1056	/* Guest - real mode */
1057	#define PGM_GST_TYPE PGM_TYPE_REAL
1058	#define PGM_GST_NAME(name) PGM_GST_NAME_REAL(name)
1059	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_REAL_STR(name)
1060	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_REAL_STR(name)
1061	#define PGM_BTH_NAME(name) PGM_BTH_NAME_EPT_REAL(name)
1062	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_EPT_REAL_STR(name)
1063	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_EPT_REAL_STR(name)
1064	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
1065	#include "PGMGstDefs.h"
1066	#include "PGMBth.h"
1067	#undef BTH_PGMPOOLKIND_PT_FOR_PT
1068	#undef PGM_BTH_NAME
1069	#undef PGM_BTH_NAME_RC_STR
1070	#undef PGM_BTH_NAME_R0_STR
1071	#undef PGM_GST_TYPE
1072	#undef PGM_GST_NAME
1073	#undef PGM_GST_NAME_RC_STR
1074	#undef PGM_GST_NAME_R0_STR
1075
1076	/* Guest - protected mode */
1077	#define PGM_GST_TYPE PGM_TYPE_PROT
1078	#define PGM_GST_NAME(name) PGM_GST_NAME_PROT(name)
1079	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PROT_STR(name)
1080	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PROT_STR(name)
1081	#define PGM_BTH_NAME(name) PGM_BTH_NAME_EPT_PROT(name)
1082	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_EPT_PROT_STR(name)
1083	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_EPT_PROT_STR(name)
1084	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PHYS
1085	#include "PGMGstDefs.h"
1086	#include "PGMBth.h"
1087	#undef BTH_PGMPOOLKIND_PT_FOR_PT
1088	#undef PGM_BTH_NAME
1089	#undef PGM_BTH_NAME_RC_STR
1090	#undef PGM_BTH_NAME_R0_STR
1091	#undef PGM_GST_TYPE
1092	#undef PGM_GST_NAME
1093	#undef PGM_GST_NAME_RC_STR
1094	#undef PGM_GST_NAME_R0_STR
1095
1096	/* Guest - 32-bit mode */
1097	#define PGM_GST_TYPE PGM_TYPE_32BIT
1098	#define PGM_GST_NAME(name) PGM_GST_NAME_32BIT(name)
1099	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_32BIT_STR(name)
1100	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_32BIT_STR(name)
1101	#define PGM_BTH_NAME(name) PGM_BTH_NAME_EPT_32BIT(name)
1102	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_EPT_32BIT_STR(name)
1103	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_EPT_32BIT_STR(name)
1104	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_32BIT_PT
1105	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_32BIT_4MB
1106	#include "PGMGstDefs.h"
1107	#include "PGMBth.h"
1108	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
1109	#undef BTH_PGMPOOLKIND_PT_FOR_PT
1110	#undef PGM_BTH_NAME
1111	#undef PGM_BTH_NAME_RC_STR
1112	#undef PGM_BTH_NAME_R0_STR
1113	#undef PGM_GST_TYPE
1114	#undef PGM_GST_NAME
1115	#undef PGM_GST_NAME_RC_STR
1116	#undef PGM_GST_NAME_R0_STR
1117
1118	/* Guest - PAE mode */
1119	#define PGM_GST_TYPE PGM_TYPE_PAE
1120	#define PGM_GST_NAME(name) PGM_GST_NAME_PAE(name)
1121	#define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_PAE_STR(name)
1122	#define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_PAE_STR(name)
1123	#define PGM_BTH_NAME(name) PGM_BTH_NAME_EPT_PAE(name)
1124	#define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_EPT_PAE_STR(name)
1125	#define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_EPT_PAE_STR(name)
1126	#define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
1127	#define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
1128	#include "PGMGstDefs.h"
1129	#include "PGMBth.h"
1130	#undef BTH_PGMPOOLKIND_PT_FOR_BIG
1131	#undef BTH_PGMPOOLKIND_PT_FOR_PT
1132	#undef PGM_BTH_NAME
1133	#undef PGM_BTH_NAME_RC_STR
1134	#undef PGM_BTH_NAME_R0_STR
1135	#undef PGM_GST_TYPE
1136	#undef PGM_GST_NAME
1137	#undef PGM_GST_NAME_RC_STR
1138	#undef PGM_GST_NAME_R0_STR
1139
1140	#ifdef VBOX_WITH_64_BITS_GUESTS
1141	/* Guest - AMD64 mode */
1142	# define PGM_GST_TYPE PGM_TYPE_AMD64
1143	# define PGM_GST_NAME(name) PGM_GST_NAME_AMD64(name)
1144	# define PGM_GST_NAME_RC_STR(name) PGM_GST_NAME_RC_AMD64_STR(name)
1145	# define PGM_GST_NAME_R0_STR(name) PGM_GST_NAME_R0_AMD64_STR(name)
1146	# define PGM_BTH_NAME(name) PGM_BTH_NAME_EPT_AMD64(name)
1147	# define PGM_BTH_NAME_RC_STR(name) PGM_BTH_NAME_RC_EPT_AMD64_STR(name)
1148	# define PGM_BTH_NAME_R0_STR(name) PGM_BTH_NAME_R0_EPT_AMD64_STR(name)
1149	# define BTH_PGMPOOLKIND_PT_FOR_PT PGMPOOLKIND_PAE_PT_FOR_PAE_PT
1150	# define BTH_PGMPOOLKIND_PT_FOR_BIG PGMPOOLKIND_PAE_PT_FOR_PAE_2MB
1151	# include "PGMGstDefs.h"
1152	# include "PGMBth.h"
1153	# undef BTH_PGMPOOLKIND_PT_FOR_BIG
1154	# undef BTH_PGMPOOLKIND_PT_FOR_PT
1155	# undef PGM_BTH_NAME
1156	# undef PGM_BTH_NAME_RC_STR
1157	# undef PGM_BTH_NAME_R0_STR
1158	# undef PGM_GST_TYPE
1159	# undef PGM_GST_NAME
1160	# undef PGM_GST_NAME_RC_STR
1161	# undef PGM_GST_NAME_R0_STR
1162	#endif /* VBOX_WITH_64_BITS_GUESTS */
1163
1164	#undef PGM_SHW_TYPE
1165	#undef PGM_SHW_NAME
1166	#undef PGM_SHW_NAME_RC_STR
1167	#undef PGM_SHW_NAME_R0_STR
1168
1169
1170
1171	/**
1172	* Initiates the paging of VM.
1173	*
1174	* @returns VBox status code.
1175	* @param pVM Pointer to VM structure.
1176	*/
1177	VMMR3DECL(int) PGMR3Init(PVM pVM)
1178	{
1179	LogFlow(("PGMR3Init:\n"));
1180	PCFGMNODE pCfgPGM = CFGMR3GetChild(CFGMR3GetRoot(pVM), "/PGM");
1181	int rc;
1182
1183	/*
1184	* Assert alignment and sizes.
1185	*/
1186	AssertCompile(sizeof(pVM->pgm.s) <= sizeof(pVM->pgm.padding));
1187	AssertCompile(sizeof(pVM->aCpus[0].pgm.s) <= sizeof(pVM->aCpus[0].pgm.padding));
1188	AssertCompileMemberAlignment(PGM, CritSect, sizeof(uintptr_t));
1189
1190	/*
1191	* Init the structure.
1192	*/
1193	#ifdef PGM_WITHOUT_MAPPINGS
1194	pVM->pgm.s.fMappingsDisabled = true;
1195	#endif
1196	pVM->pgm.s.offVM = RT_OFFSETOF(VM, pgm.s);
1197	pVM->pgm.s.offVCpuPGM = RT_OFFSETOF(VMCPU, pgm.s);
1198
1199	/* Init the per-CPU part. */
1200	for (VMCPUID idCpu = 0; idCpu < pVM->cCpus; idCpu++)
1201	{
1202	PVMCPU pVCpu = &pVM->aCpus[idCpu];
1203	PPGMCPU pPGM = &pVCpu->pgm.s;
1204
1205	pPGM->offVM = (uintptr_t)&pVCpu->pgm.s - (uintptr_t)pVM;
1206	pPGM->offVCpu = RT_OFFSETOF(VMCPU, pgm.s);
1207	pPGM->offPGM = (uintptr_t)&pVCpu->pgm.s - (uintptr_t)&pVM->pgm.s;
1208
1209	pPGM->enmShadowMode = PGMMODE_INVALID;
1210	pPGM->enmGuestMode = PGMMODE_INVALID;
1211
1212	pPGM->GCPhysCR3 = NIL_RTGCPHYS;
1213
1214	pPGM->pGst32BitPdR3 = NULL;
1215	pPGM->pGstPaePdptR3 = NULL;
1216	pPGM->pGstAmd64Pml4R3 = NULL;
1217	#ifndef VBOX_WITH_2X_4GB_ADDR_SPACE
1218	pPGM->pGst32BitPdR0 = NIL_RTR0PTR;
1219	pPGM->pGstPaePdptR0 = NIL_RTR0PTR;
1220	pPGM->pGstAmd64Pml4R0 = NIL_RTR0PTR;
1221	#endif
1222	pPGM->pGst32BitPdRC = NIL_RTRCPTR;
1223	pPGM->pGstPaePdptRC = NIL_RTRCPTR;
1224	for (unsigned i = 0; i < RT_ELEMENTS(pVCpu->pgm.s.apGstPaePDsR3); i++)
1225	{
1226	pPGM->apGstPaePDsR3[i] = NULL;
1227	#ifndef VBOX_WITH_2X_4GB_ADDR_SPACE
1228	pPGM->apGstPaePDsR0[i] = NIL_RTR0PTR;
1229	#endif
1230	pPGM->apGstPaePDsRC[i] = NIL_RTRCPTR;
1231	pPGM->aGCPhysGstPaePDs[i] = NIL_RTGCPHYS;
1232	pPGM->aGCPhysGstPaePDsMonitored[i] = NIL_RTGCPHYS;
1233	}
1234
1235	pPGM->fA20Enabled = true;
1236	}
1237
1238	pVM->pgm.s.enmHostMode = SUPPAGINGMODE_INVALID;
1239	pVM->pgm.s.GCPhys4MBPSEMask = RT_BIT_64(32) - 1; /* default; checked later */
1240	pVM->pgm.s.GCPtrPrevRamRangeMapping = MM_HYPER_AREA_ADDRESS;
1241
1242	rc = CFGMR3QueryBoolDef(CFGMR3GetRoot(pVM), "RamPreAlloc", &pVM->pgm.s.fRamPreAlloc,
1243	#ifdef VBOX_WITH_PREALLOC_RAM_BY_DEFAULT
1244	true
1245	#else
1246	false
1247	#endif
1248	);
1249	AssertLogRelRCReturn(rc, rc);
1250
1251	#ifdef PGM_WITH_LARGE_ADDRESS_SPACE_ON_32_BIT_HOST
1252	rc = CFGMR3QueryU32Def(pCfgPGM, "MaxRing3Chunks", &pVM->pgm.s.ChunkR3Map.cMax, _1G / GMM_CHUNK_SIZE);
1253	#else
1254	rc = CFGMR3QueryU32Def(pCfgPGM, "MaxRing3Chunks", &pVM->pgm.s.ChunkR3Map.cMax, UINT32_MAX);
1255	#endif
1256	AssertLogRelRCReturn(rc, rc);
1257	for (uint32_t i = 0; i < RT_ELEMENTS(pVM->pgm.s.ChunkR3Map.Tlb.aEntries); i++)
1258	pVM->pgm.s.ChunkR3Map.Tlb.aEntries[i].idChunk = NIL_GMM_CHUNKID;
1259
1260	/*
1261	* Get the configured RAM size - to estimate saved state size.
1262	*/
1263	uint64_t cbRam;
1264	rc = CFGMR3QueryU64(CFGMR3GetRoot(pVM), "RamSize", &cbRam);
1265	if (rc == VERR_CFGM_VALUE_NOT_FOUND)
1266	cbRam = 0;
1267	else if (RT_SUCCESS(rc))
1268	{
1269	if (cbRam < PAGE_SIZE)
1270	cbRam = 0;
1271	cbRam = RT_ALIGN_64(cbRam, PAGE_SIZE);
1272	}
1273	else
1274	{
1275	AssertMsgFailed(("Configuration error: Failed to query integer \"RamSize\", rc=%Rrc.\n", rc));
1276	return rc;
1277	}
1278
1279	/*
1280	* Check for PCI pass-through.
1281	*/
1282	rc = CFGMR3QueryBoolDef(pCfgPGM, "PciPassThrough", &pVM->pgm.s.fPciPassthrough, false);
1283	AssertMsgRCReturn(rc, ("Configuration error: Failed to query integer \"PciPassThrough\", rc=%Rrc.\n", rc), rc);
1284	AssertLogRelReturn(!pVM->pgm.s.fPciPassthrough \|\| pVM->pgm.s.fRamPreAlloc, VERR_INVALID_PARAMETER);
1285
1286	#ifdef VBOX_WITH_STATISTICS
1287	/*
1288	* Allocate memory for the statistics before someone tries to use them.
1289	*/
1290	size_t cbTotalStats = RT_ALIGN_Z(sizeof(PGMSTATS), 64) + RT_ALIGN_Z(sizeof(PGMCPUSTATS), 64) * pVM->cCpus;
1291	void *pv;
1292	rc = MMHyperAlloc(pVM, RT_ALIGN_Z(cbTotalStats, PAGE_SIZE), PAGE_SIZE, MM_TAG_PGM, &pv);
1293	AssertRCReturn(rc, rc);
1294
1295	pVM->pgm.s.pStatsR3 = (PGMSTATS *)pv;
1296	pVM->pgm.s.pStatsR0 = MMHyperCCToR0(pVM, pv);
1297	pVM->pgm.s.pStatsRC = MMHyperCCToRC(pVM, pv);
1298	pv = (uint8_t *)pv + RT_ALIGN_Z(sizeof(PGMSTATS), 64);
1299
1300	for (VMCPUID iCpu = 0; iCpu < pVM->cCpus; iCpu++)
1301	{
1302	pVM->aCpus[iCpu].pgm.s.pStatsR3 = (PGMCPUSTATS *)pv;
1303	pVM->aCpus[iCpu].pgm.s.pStatsR0 = MMHyperCCToR0(pVM, pv);
1304	pVM->aCpus[iCpu].pgm.s.pStatsRC = MMHyperCCToRC(pVM, pv);
1305
1306	pv = (uint8_t *)pv + RT_ALIGN_Z(sizeof(PGMCPUSTATS), 64);
1307	}
1308	#endif /* VBOX_WITH_STATISTICS */
1309
1310	/*
1311	* Register callbacks, string formatters and the saved state data unit.
1312	*/
1313	#ifdef VBOX_STRICT
1314	VMR3AtStateRegister(pVM, pgmR3ResetNoMorePhysWritesFlag, NULL);
1315	#endif
1316	PGMRegisterStringFormatTypes();
1317
1318	rc = pgmR3InitSavedState(pVM, cbRam);
1319	if (RT_FAILURE(rc))
1320	return rc;
1321
1322	/*
1323	* Initialize the PGM critical section and flush the phys TLBs
1324	*/
1325	rc = PDMR3CritSectInit(pVM, &pVM->pgm.s.CritSect, RT_SRC_POS, "PGM");
1326	AssertRCReturn(rc, rc);
1327
1328	PGMR3PhysChunkInvalidateTLB(pVM);
1329	pgmPhysInvalidatePageMapTLB(pVM);
1330
1331	/*
1332	* For the time being we sport a full set of handy pages in addition to the base
1333	* memory to simplify things.
1334	*/
1335	rc = MMR3ReserveHandyPages(pVM, RT_ELEMENTS(pVM->pgm.s.aHandyPages)); /** @todo this should be changed to PGM_HANDY_PAGES_MIN but this needs proper testing... */
1336	AssertRCReturn(rc, rc);
1337
1338	/*
1339	* Trees
1340	*/
1341	rc = MMHyperAlloc(pVM, sizeof(PGMTREES), 0, MM_TAG_PGM, (void **)&pVM->pgm.s.pTreesR3);
1342	if (RT_SUCCESS(rc))
1343	{
1344	pVM->pgm.s.pTreesR0 = MMHyperR3ToR0(pVM, pVM->pgm.s.pTreesR3);
1345	pVM->pgm.s.pTreesRC = MMHyperR3ToRC(pVM, pVM->pgm.s.pTreesR3);
1346
1347	/*
1348	* Allocate the zero page.
1349	*/
1350	rc = MMHyperAlloc(pVM, PAGE_SIZE, PAGE_SIZE, MM_TAG_PGM, &pVM->pgm.s.pvZeroPgR3);
1351	}
1352	if (RT_SUCCESS(rc))
1353	{
1354	pVM->pgm.s.pvZeroPgRC = MMHyperR3ToRC(pVM, pVM->pgm.s.pvZeroPgR3);
1355	pVM->pgm.s.pvZeroPgR0 = MMHyperR3ToR0(pVM, pVM->pgm.s.pvZeroPgR3);
1356	pVM->pgm.s.HCPhysZeroPg = MMR3HyperHCVirt2HCPhys(pVM, pVM->pgm.s.pvZeroPgR3);
1357	AssertRelease(pVM->pgm.s.HCPhysZeroPg != NIL_RTHCPHYS);
1358
1359	/*
1360	* Allocate the invalid MMIO page.
1361	* (The invalid bits in HCPhysInvMmioPg are set later on init complete.)
1362	*/
1363	rc = MMHyperAlloc(pVM, PAGE_SIZE, PAGE_SIZE, MM_TAG_PGM, &pVM->pgm.s.pvMmioPgR3);
1364	}
1365	if (RT_SUCCESS(rc))
1366	{
1367	ASMMemFill32(pVM->pgm.s.pvMmioPgR3, PAGE_SIZE, 0xfeedface);
1368	pVM->pgm.s.HCPhysMmioPg = MMR3HyperHCVirt2HCPhys(pVM, pVM->pgm.s.pvMmioPgR3);
1369	AssertRelease(pVM->pgm.s.HCPhysMmioPg != NIL_RTHCPHYS);
1370	pVM->pgm.s.HCPhysInvMmioPg = pVM->pgm.s.HCPhysMmioPg;
1371
1372	/*
1373	* Init the paging.
1374	*/
1375	rc = pgmR3InitPaging(pVM);
1376	}
1377	if (RT_SUCCESS(rc))
1378	{
1379	/*
1380	* Init the page pool.
1381	*/
1382	rc = pgmR3PoolInit(pVM);
1383	}
1384	if (RT_SUCCESS(rc))
1385	{
1386	for (VMCPUID i = 0; i < pVM->cCpus; i++)
1387	{
1388	PVMCPU pVCpu = &pVM->aCpus[i];
1389	rc = PGMR3ChangeMode(pVM, pVCpu, PGMMODE_REAL);
1390	if (RT_FAILURE(rc))
1391	break;
1392	}
1393	}
1394
1395	if (RT_SUCCESS(rc))
1396	{
1397	/*
1398	* Info & statistics
1399	*/
1400	DBGFR3InfoRegisterInternal(pVM, "mode",
1401	"Shows the current paging mode. "
1402	"Recognizes 'all', 'guest', 'shadow' and 'host' as arguments, defaulting to 'all' if nothing is given.",
1403	pgmR3InfoMode);
1404	DBGFR3InfoRegisterInternal(pVM, "pgmcr3",
1405	"Dumps all the entries in the top level paging table. No arguments.",
1406	pgmR3InfoCr3);
1407	DBGFR3InfoRegisterInternal(pVM, "phys",
1408	"Dumps all the physical address ranges. No arguments.",
1409	pgmR3PhysInfo);
1410	DBGFR3InfoRegisterInternal(pVM, "handlers",
1411	"Dumps physical, virtual and hyper virtual handlers. "
1412	"Pass 'phys', 'virt', 'hyper' as argument if only one kind is wanted."
1413	"Add 'nost' if the statistics are unwanted, use together with 'all' or explicit selection.",
1414	pgmR3InfoHandlers);
1415	DBGFR3InfoRegisterInternal(pVM, "mappings",
1416	"Dumps guest mappings.",
1417	pgmR3MapInfo);
1418
1419	pgmR3InitStats(pVM);
1420
1421	#ifdef VBOX_WITH_DEBUGGER
1422	/*
1423	* Debugger commands.
1424	*/
1425	static bool s_fRegisteredCmds = false;
1426	if (!s_fRegisteredCmds)
1427	{
1428	int rc2 = DBGCRegisterCommands(&g_aCmds[0], RT_ELEMENTS(g_aCmds));
1429	if (RT_SUCCESS(rc2))
1430	s_fRegisteredCmds = true;
1431	}
1432	#endif
1433	return VINF_SUCCESS;
1434	}
1435
1436	/* Almost no cleanup necessary, MM frees all memory. */
1437	PDMR3CritSectDelete(&pVM->pgm.s.CritSect);
1438
1439	return rc;
1440	}
1441
1442
1443	/**
1444	* Init paging.
1445	*
1446	* Since we need to check what mode the host is operating in before we can choose
1447	* the right paging functions for the host we have to delay this until R0 has
1448	* been initialized.
1449	*
1450	* @returns VBox status code.
1451	* @param pVM VM handle.
1452	*/
1453	static int pgmR3InitPaging(PVM pVM)
1454	{
1455	/*
1456	* Force a recalculation of modes and switcher so everyone gets notified.
1457	*/
1458	for (VMCPUID i = 0; i < pVM->cCpus; i++)
1459	{
1460	PVMCPU pVCpu = &pVM->aCpus[i];
1461
1462	pVCpu->pgm.s.enmShadowMode = PGMMODE_INVALID;
1463	pVCpu->pgm.s.enmGuestMode = PGMMODE_INVALID;
1464	}
1465
1466	pVM->pgm.s.enmHostMode = SUPPAGINGMODE_INVALID;
1467
1468	/*
1469	* Allocate static mapping space for whatever the cr3 register
1470	* points to and in the case of PAE mode to the 4 PDs.
1471	*/
1472	int rc = MMR3HyperReserve(pVM, PAGE_SIZE * 5, "CR3 mapping", &pVM->pgm.s.GCPtrCR3Mapping);
1473	if (RT_FAILURE(rc))
1474	{
1475	AssertMsgFailed(("Failed to reserve two pages for cr mapping in HMA, rc=%Rrc\n", rc));
1476	return rc;
1477	}
1478	MMR3HyperReserve(pVM, PAGE_SIZE, "fence", NULL);
1479
1480	/*
1481	* Allocate pages for the three possible intermediate contexts
1482	* (AMD64, PAE and plain 32-Bit). We maintain all three contexts
1483	* for the sake of simplicity. The AMD64 uses the PAE for the
1484	* lower levels, making the total number of pages 11 (3 + 7 + 1).
1485	*
1486	* We assume that two page tables will be enought for the core code
1487	* mappings (HC virtual and identity).
1488	*/
1489	pVM->pgm.s.pInterPD = (PX86PD)MMR3PageAllocLow(pVM); AssertReturn(pVM->pgm.s.pInterPD, VERR_NO_PAGE_MEMORY);
1490	pVM->pgm.s.apInterPTs[0] = (PX86PT)MMR3PageAllocLow(pVM); AssertReturn(pVM->pgm.s.apInterPTs[0], VERR_NO_PAGE_MEMORY);
1491	pVM->pgm.s.apInterPTs[1] = (PX86PT)MMR3PageAllocLow(pVM); AssertReturn(pVM->pgm.s.apInterPTs[1], VERR_NO_PAGE_MEMORY);
1492	pVM->pgm.s.apInterPaePTs[0] = (PX86PTPAE)MMR3PageAlloc(pVM); AssertReturn(pVM->pgm.s.apInterPaePTs[0], VERR_NO_PAGE_MEMORY);
1493	pVM->pgm.s.apInterPaePTs[1] = (PX86PTPAE)MMR3PageAlloc(pVM); AssertReturn(pVM->pgm.s.apInterPaePTs[1], VERR_NO_PAGE_MEMORY);
1494	pVM->pgm.s.apInterPaePDs[0] = (PX86PDPAE)MMR3PageAlloc(pVM); AssertReturn(pVM->pgm.s.apInterPaePDs[0], VERR_NO_PAGE_MEMORY);
1495	pVM->pgm.s.apInterPaePDs[1] = (PX86PDPAE)MMR3PageAlloc(pVM); AssertReturn(pVM->pgm.s.apInterPaePDs[1], VERR_NO_PAGE_MEMORY);
1496	pVM->pgm.s.apInterPaePDs[2] = (PX86PDPAE)MMR3PageAlloc(pVM); AssertReturn(pVM->pgm.s.apInterPaePDs[2], VERR_NO_PAGE_MEMORY);
1497	pVM->pgm.s.apInterPaePDs[3] = (PX86PDPAE)MMR3PageAlloc(pVM); AssertReturn(pVM->pgm.s.apInterPaePDs[3], VERR_NO_PAGE_MEMORY);
1498	pVM->pgm.s.pInterPaePDPT = (PX86PDPT)MMR3PageAllocLow(pVM); AssertReturn(pVM->pgm.s.pInterPaePDPT, VERR_NO_PAGE_MEMORY);
1499	pVM->pgm.s.pInterPaePDPT64 = (PX86PDPT)MMR3PageAllocLow(pVM); AssertReturn(pVM->pgm.s.pInterPaePDPT64, VERR_NO_PAGE_MEMORY);
1500	pVM->pgm.s.pInterPaePML4 = (PX86PML4)MMR3PageAllocLow(pVM); AssertReturn(pVM->pgm.s.pInterPaePML4, VERR_NO_PAGE_MEMORY);
1501
1502	pVM->pgm.s.HCPhysInterPD = MMPage2Phys(pVM, pVM->pgm.s.pInterPD);
1503	AssertRelease(pVM->pgm.s.HCPhysInterPD != NIL_RTHCPHYS && !(pVM->pgm.s.HCPhysInterPD & PAGE_OFFSET_MASK));
1504	pVM->pgm.s.HCPhysInterPaePDPT = MMPage2Phys(pVM, pVM->pgm.s.pInterPaePDPT);
1505	AssertRelease(pVM->pgm.s.HCPhysInterPaePDPT != NIL_RTHCPHYS && !(pVM->pgm.s.HCPhysInterPaePDPT & PAGE_OFFSET_MASK));
1506	pVM->pgm.s.HCPhysInterPaePML4 = MMPage2Phys(pVM, pVM->pgm.s.pInterPaePML4);
1507	AssertRelease(pVM->pgm.s.HCPhysInterPaePML4 != NIL_RTHCPHYS && !(pVM->pgm.s.HCPhysInterPaePML4 & PAGE_OFFSET_MASK) && pVM->pgm.s.HCPhysInterPaePML4 < 0xffffffff);
1508
1509	/*
1510	* Initialize the pages, setting up the PML4 and PDPT for repetitive 4GB action.
1511	*/
1512	ASMMemZeroPage(pVM->pgm.s.pInterPD);
1513	ASMMemZeroPage(pVM->pgm.s.apInterPTs[0]);
1514	ASMMemZeroPage(pVM->pgm.s.apInterPTs[1]);
1515
1516	ASMMemZeroPage(pVM->pgm.s.apInterPaePTs[0]);
1517	ASMMemZeroPage(pVM->pgm.s.apInterPaePTs[1]);
1518
1519	ASMMemZeroPage(pVM->pgm.s.pInterPaePDPT);
1520	for (unsigned i = 0; i < RT_ELEMENTS(pVM->pgm.s.apInterPaePDs); i++)
1521	{
1522	ASMMemZeroPage(pVM->pgm.s.apInterPaePDs[i]);
1523	pVM->pgm.s.pInterPaePDPT->a[i].u = X86_PDPE_P \| PGM_PLXFLAGS_PERMANENT
1524	\| MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[i]);
1525	}
1526
1527	for (unsigned i = 0; i < RT_ELEMENTS(pVM->pgm.s.pInterPaePDPT64->a); i++)
1528	{
1529	const unsigned iPD = i % RT_ELEMENTS(pVM->pgm.s.apInterPaePDs);
1530	pVM->pgm.s.pInterPaePDPT64->a[i].u = X86_PDPE_P \| X86_PDPE_RW \| X86_PDPE_US \| X86_PDPE_A \| PGM_PLXFLAGS_PERMANENT
1531	\| MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[iPD]);
1532	}
1533
1534	RTHCPHYS HCPhysInterPaePDPT64 = MMPage2Phys(pVM, pVM->pgm.s.pInterPaePDPT64);
1535	for (unsigned i = 0; i < RT_ELEMENTS(pVM->pgm.s.pInterPaePML4->a); i++)
1536	pVM->pgm.s.pInterPaePML4->a[i].u = X86_PML4E_P \| X86_PML4E_RW \| X86_PML4E_US \| X86_PML4E_A \| PGM_PLXFLAGS_PERMANENT
1537	\| HCPhysInterPaePDPT64;
1538
1539	/*
1540	* Initialize paging workers and mode from current host mode
1541	* and the guest running in real mode.
1542	*/
1543	pVM->pgm.s.enmHostMode = SUPR3GetPagingMode();
1544	switch (pVM->pgm.s.enmHostMode)
1545	{
1546	case SUPPAGINGMODE_32_BIT:
1547	case SUPPAGINGMODE_32_BIT_GLOBAL:
1548	case SUPPAGINGMODE_PAE:
1549	case SUPPAGINGMODE_PAE_GLOBAL:
1550	case SUPPAGINGMODE_PAE_NX:
1551	case SUPPAGINGMODE_PAE_GLOBAL_NX:
1552	break;
1553
1554	case SUPPAGINGMODE_AMD64:
1555	case SUPPAGINGMODE_AMD64_GLOBAL:
1556	case SUPPAGINGMODE_AMD64_NX:
1557	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
1558	#ifndef VBOX_WITH_HYBRID_32BIT_KERNEL
1559	if (ARCH_BITS != 64)
1560	{
1561	AssertMsgFailed(("Host mode %d (64-bit) is not supported by non-64bit builds\n", pVM->pgm.s.enmHostMode));
1562	LogRel(("Host mode %d (64-bit) is not supported by non-64bit builds\n", pVM->pgm.s.enmHostMode));
1563	return VERR_PGM_UNSUPPORTED_HOST_PAGING_MODE;
1564	}
1565	#endif
1566	break;
1567	default:
1568	AssertMsgFailed(("Host mode %d is not supported\n", pVM->pgm.s.enmHostMode));
1569	return VERR_PGM_UNSUPPORTED_HOST_PAGING_MODE;
1570	}
1571	rc = pgmR3ModeDataInit(pVM, false /* don't resolve GC and R0 syms yet */);
1572	if (RT_SUCCESS(rc))
1573	{
1574	LogFlow(("pgmR3InitPaging: returns successfully\n"));
1575	#if HC_ARCH_BITS == 64
1576	LogRel(("Debug: HCPhysInterPD=%RHp HCPhysInterPaePDPT=%RHp HCPhysInterPaePML4=%RHp\n",
1577	pVM->pgm.s.HCPhysInterPD, pVM->pgm.s.HCPhysInterPaePDPT, pVM->pgm.s.HCPhysInterPaePML4));
1578	LogRel(("Debug: apInterPTs={%RHp,%RHp} apInterPaePTs={%RHp,%RHp} apInterPaePDs={%RHp,%RHp,%RHp,%RHp} pInterPaePDPT64=%RHp\n",
1579	MMPage2Phys(pVM, pVM->pgm.s.apInterPTs[0]), MMPage2Phys(pVM, pVM->pgm.s.apInterPTs[1]),
1580	MMPage2Phys(pVM, pVM->pgm.s.apInterPaePTs[0]), MMPage2Phys(pVM, pVM->pgm.s.apInterPaePTs[1]),
1581	MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[0]), MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[1]), MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[2]), MMPage2Phys(pVM, pVM->pgm.s.apInterPaePDs[3]),
1582	MMPage2Phys(pVM, pVM->pgm.s.pInterPaePDPT64)));
1583	#endif
1584	return VINF_SUCCESS;
1585	}
1586
1587	LogFlow(("pgmR3InitPaging: returns %Rrc\n", rc));
1588	return rc;
1589	}
1590
1591
1592	/**
1593	* Init statistics
1594	* @returns VBox status code.
1595	*/
1596	static int pgmR3InitStats(PVM pVM)
1597	{
1598	PPGM pPGM = &pVM->pgm.s;
1599	int rc;
1600
1601	/*
1602	* Release statistics.
1603	*/
1604	/* Common - misc variables */
1605	STAM_REL_REG(pVM, &pPGM->cAllPages, STAMTYPE_U32, "/PGM/Page/cAllPages", STAMUNIT_COUNT, "The total number of pages.");
1606	STAM_REL_REG(pVM, &pPGM->cPrivatePages, STAMTYPE_U32, "/PGM/Page/cPrivatePages", STAMUNIT_COUNT, "The number of private pages.");
1607	STAM_REL_REG(pVM, &pPGM->cSharedPages, STAMTYPE_U32, "/PGM/Page/cSharedPages", STAMUNIT_COUNT, "The number of shared pages.");
1608	STAM_REL_REG(pVM, &pPGM->cReusedSharedPages, STAMTYPE_U32, "/PGM/Page/cReusedSharedPages", STAMUNIT_COUNT, "The number of reused shared pages.");
1609	STAM_REL_REG(pVM, &pPGM->cZeroPages, STAMTYPE_U32, "/PGM/Page/cZeroPages", STAMUNIT_COUNT, "The number of zero backed pages.");
1610	STAM_REL_REG(pVM, &pPGM->cPureMmioPages, STAMTYPE_U32, "/PGM/Page/cPureMmioPages", STAMUNIT_COUNT, "The number of pure MMIO pages.");
1611	STAM_REL_REG(pVM, &pPGM->cMonitoredPages, STAMTYPE_U32, "/PGM/Page/cMonitoredPages", STAMUNIT_COUNT, "The number of write monitored pages.");
1612	STAM_REL_REG(pVM, &pPGM->cWrittenToPages, STAMTYPE_U32, "/PGM/Page/cWrittenToPages", STAMUNIT_COUNT, "The number of previously write monitored pages that have been written to.");
1613	STAM_REL_REG(pVM, &pPGM->cWriteLockedPages, STAMTYPE_U32, "/PGM/Page/cWriteLockedPages", STAMUNIT_COUNT, "The number of write(/read) locked pages.");
1614	STAM_REL_REG(pVM, &pPGM->cReadLockedPages, STAMTYPE_U32, "/PGM/Page/cReadLockedPages", STAMUNIT_COUNT, "The number of read (only) locked pages.");
1615	STAM_REL_REG(pVM, &pPGM->cBalloonedPages, STAMTYPE_U32, "/PGM/Page/cBalloonedPages", STAMUNIT_COUNT, "The number of ballooned pages.");
1616	STAM_REL_REG(pVM, &pPGM->cHandyPages, STAMTYPE_U32, "/PGM/Page/cHandyPages", STAMUNIT_COUNT, "The number of handy pages (not included in cAllPages).");
1617	STAM_REL_REG(pVM, &pPGM->cLargePages, STAMTYPE_U32, "/PGM/Page/cLargePages", STAMUNIT_COUNT, "The number of large pages allocated (includes disabled).");
1618	STAM_REL_REG(pVM, &pPGM->cLargePagesDisabled, STAMTYPE_U32, "/PGM/Page/cLargePagesDisabled", STAMUNIT_COUNT, "The number of disabled large pages.");
1619	STAM_REL_REG(pVM, &pPGM->cRelocations, STAMTYPE_COUNTER, "/PGM/cRelocations", STAMUNIT_OCCURENCES,"Number of hypervisor relocations.");
1620	STAM_REL_REG(pVM, &pPGM->ChunkR3Map.c, STAMTYPE_U32, "/PGM/ChunkR3Map/c", STAMUNIT_COUNT, "Number of mapped chunks.");
1621	STAM_REL_REG(pVM, &pPGM->ChunkR3Map.cMax, STAMTYPE_U32, "/PGM/ChunkR3Map/cMax", STAMUNIT_COUNT, "Maximum number of mapped chunks.");
1622	STAM_REL_REG(pVM, &pPGM->cMappedChunks, STAMTYPE_U32, "/PGM/ChunkR3Map/Mapped", STAMUNIT_COUNT, "Number of times we mapped a chunk.");
1623	STAM_REL_REG(pVM, &pPGM->cUnmappedChunks, STAMTYPE_U32, "/PGM/ChunkR3Map/Unmapped", STAMUNIT_COUNT, "Number of times we unmapped a chunk.");
1624
1625	STAM_REL_REG(pVM, &pPGM->StatLargePageReused, STAMTYPE_COUNTER, "/PGM/LargePage/Reused", STAMUNIT_OCCURENCES, "The number of times we've reused a large page.");
1626	STAM_REL_REG(pVM, &pPGM->StatLargePageRefused, STAMTYPE_COUNTER, "/PGM/LargePage/Refused", STAMUNIT_OCCURENCES, "The number of times we couldn't use a large page.");
1627	STAM_REL_REG(pVM, &pPGM->StatLargePageRecheck, STAMTYPE_COUNTER, "/PGM/LargePage/Recheck", STAMUNIT_OCCURENCES, "The number of times we've rechecked a disabled large page.");
1628
1629	/* Live save */
1630	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.fActive, STAMTYPE_U8, "/PGM/LiveSave/fActive", STAMUNIT_COUNT, "Active or not.");
1631	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.cIgnoredPages, STAMTYPE_U32, "/PGM/LiveSave/cIgnoredPages", STAMUNIT_COUNT, "The number of ignored pages in the RAM ranges (i.e. MMIO, MMIO2 and ROM).");
1632	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.cDirtyPagesLong, STAMTYPE_U32, "/PGM/LiveSave/cDirtyPagesLong", STAMUNIT_COUNT, "Longer term dirty page average.");
1633	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.cDirtyPagesShort, STAMTYPE_U32, "/PGM/LiveSave/cDirtyPagesShort", STAMUNIT_COUNT, "Short term dirty page average.");
1634	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.cPagesPerSecond, STAMTYPE_U32, "/PGM/LiveSave/cPagesPerSecond", STAMUNIT_COUNT, "Pages per second.");
1635	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.cSavedPages, STAMTYPE_U64, "/PGM/LiveSave/cSavedPages", STAMUNIT_COUNT, "The total number of saved pages.");
1636	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Ram.cReadyPages, STAMTYPE_U32, "/PGM/LiveSave/Ram/cReadPages", STAMUNIT_COUNT, "RAM: Ready pages.");
1637	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Ram.cDirtyPages, STAMTYPE_U32, "/PGM/LiveSave/Ram/cDirtyPages", STAMUNIT_COUNT, "RAM: Dirty pages.");
1638	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Ram.cZeroPages, STAMTYPE_U32, "/PGM/LiveSave/Ram/cZeroPages", STAMUNIT_COUNT, "RAM: Ready zero pages.");
1639	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Ram.cMonitoredPages, STAMTYPE_U32, "/PGM/LiveSave/Ram/cMonitoredPages", STAMUNIT_COUNT, "RAM: Write monitored pages.");
1640	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Rom.cReadyPages, STAMTYPE_U32, "/PGM/LiveSave/Rom/cReadPages", STAMUNIT_COUNT, "ROM: Ready pages.");
1641	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Rom.cDirtyPages, STAMTYPE_U32, "/PGM/LiveSave/Rom/cDirtyPages", STAMUNIT_COUNT, "ROM: Dirty pages.");
1642	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Rom.cZeroPages, STAMTYPE_U32, "/PGM/LiveSave/Rom/cZeroPages", STAMUNIT_COUNT, "ROM: Ready zero pages.");
1643	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Rom.cMonitoredPages, STAMTYPE_U32, "/PGM/LiveSave/Rom/cMonitoredPages", STAMUNIT_COUNT, "ROM: Write monitored pages.");
1644	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Mmio2.cReadyPages, STAMTYPE_U32, "/PGM/LiveSave/Mmio2/cReadPages", STAMUNIT_COUNT, "MMIO2: Ready pages.");
1645	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Mmio2.cDirtyPages, STAMTYPE_U32, "/PGM/LiveSave/Mmio2/cDirtyPages", STAMUNIT_COUNT, "MMIO2: Dirty pages.");
1646	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Mmio2.cZeroPages, STAMTYPE_U32, "/PGM/LiveSave/Mmio2/cZeroPages", STAMUNIT_COUNT, "MMIO2: Ready zero pages.");
1647	STAM_REL_REG_USED(pVM, &pPGM->LiveSave.Mmio2.cMonitoredPages,STAMTYPE_U32, "/PGM/LiveSave/Mmio2/cMonitoredPages",STAMUNIT_COUNT, "MMIO2: Write monitored pages.");
1648
1649	#ifdef VBOX_WITH_STATISTICS
1650
1651	# define PGM_REG_COUNTER(a, b, c) \
1652	rc = STAMR3RegisterF(pVM, a, STAMTYPE_COUNTER, STAMVISIBILITY_ALWAYS, STAMUNIT_OCCURENCES, c, b); \
1653	AssertRC(rc);
1654
1655	# define PGM_REG_COUNTER_BYTES(a, b, c) \
1656	rc = STAMR3RegisterF(pVM, a, STAMTYPE_COUNTER, STAMVISIBILITY_ALWAYS, STAMUNIT_BYTES, c, b); \
1657	AssertRC(rc);
1658
1659	# define PGM_REG_PROFILE(a, b, c) \
1660	rc = STAMR3RegisterF(pVM, a, STAMTYPE_PROFILE, STAMVISIBILITY_ALWAYS, STAMUNIT_TICKS_PER_CALL, c, b); \
1661	AssertRC(rc);
1662
1663	PGMSTATS *pStats = pVM->pgm.s.pStatsR3;
1664
1665	PGM_REG_PROFILE(&pStats->StatAllocLargePage, "/PGM/LargePage/Prof/Alloc", "Time spent by the host OS for large page allocation.");
1666	PGM_REG_PROFILE(&pStats->StatClearLargePage, "/PGM/LargePage/Prof/Clear", "Time spent clearing the newly allocated large pages.");
1667	PGM_REG_COUNTER(&pStats->StatLargePageOverflow, "/PGM/LargePage/Overflow", "The number of times allocating a large page took too long.");
1668	PGM_REG_PROFILE(&pStats->StatR3IsValidLargePage, "/PGM/LargePage/Prof/R3/IsValid", "pgmPhysIsValidLargePage profiling - R3.");
1669	PGM_REG_PROFILE(&pStats->StatRZIsValidLargePage, "/PGM/LargePage/Prof/RZ/IsValid", "pgmPhysIsValidLargePage profiling - RZ.");
1670
1671	PGM_REG_COUNTER(&pStats->StatR3DetectedConflicts, "/PGM/R3/DetectedConflicts", "The number of times PGMR3CheckMappingConflicts() detected a conflict.");
1672	PGM_REG_PROFILE(&pStats->StatR3ResolveConflict, "/PGM/R3/ResolveConflict", "pgmR3SyncPTResolveConflict() profiling (includes the entire relocation).");
1673	PGM_REG_COUNTER(&pStats->StatR3PhysRead, "/PGM/R3/Phys/Read", "The number of times PGMPhysRead was called.");
1674	PGM_REG_COUNTER_BYTES(&pStats->StatR3PhysReadBytes, "/PGM/R3/Phys/Read/Bytes", "The number of bytes read by PGMPhysRead.");
1675	PGM_REG_COUNTER(&pStats->StatR3PhysWrite, "/PGM/R3/Phys/Write", "The number of times PGMPhysWrite was called.");
1676	PGM_REG_COUNTER_BYTES(&pStats->StatR3PhysWriteBytes, "/PGM/R3/Phys/Write/Bytes", "The number of bytes written by PGMPhysWrite.");
1677	PGM_REG_COUNTER(&pStats->StatR3PhysSimpleRead, "/PGM/R3/Phys/Simple/Read", "The number of times PGMPhysSimpleReadGCPtr was called.");
1678	PGM_REG_COUNTER_BYTES(&pStats->StatR3PhysSimpleReadBytes, "/PGM/R3/Phys/Simple/Read/Bytes", "The number of bytes read by PGMPhysSimpleReadGCPtr.");
1679	PGM_REG_COUNTER(&pStats->StatR3PhysSimpleWrite, "/PGM/R3/Phys/Simple/Write", "The number of times PGMPhysSimpleWriteGCPtr was called.");
1680	PGM_REG_COUNTER_BYTES(&pStats->StatR3PhysSimpleWriteBytes, "/PGM/R3/Phys/Simple/Write/Bytes", "The number of bytes written by PGMPhysSimpleWriteGCPtr.");
1681
1682	PGM_REG_COUNTER(&pStats->StatRZChunkR3MapTlbHits, "/PGM/ChunkR3Map/TlbHitsRZ", "TLB hits.");
1683	PGM_REG_COUNTER(&pStats->StatRZChunkR3MapTlbMisses, "/PGM/ChunkR3Map/TlbMissesRZ", "TLB misses.");
1684	PGM_REG_PROFILE(&pStats->StatChunkAging, "/PGM/ChunkR3Map/Map/Aging", "Chunk aging profiling.");
1685	PGM_REG_PROFILE(&pStats->StatChunkFindCandidate, "/PGM/ChunkR3Map/Map/Find", "Chunk unmap find profiling.");
1686	PGM_REG_PROFILE(&pStats->StatChunkUnmap, "/PGM/ChunkR3Map/Map/Unmap", "Chunk unmap of address space profiling.");
1687	PGM_REG_PROFILE(&pStats->StatChunkMap, "/PGM/ChunkR3Map/Map/Map", "Chunk map of address space profiling.");
1688
1689	PGM_REG_COUNTER(&pStats->StatRZPageMapTlbHits, "/PGM/RZ/Page/MapTlbHits", "TLB hits.");
1690	PGM_REG_COUNTER(&pStats->StatRZPageMapTlbMisses, "/PGM/RZ/Page/MapTlbMisses", "TLB misses.");
1691	PGM_REG_COUNTER(&pStats->StatR3ChunkR3MapTlbHits, "/PGM/ChunkR3Map/TlbHitsR3", "TLB hits.");
1692	PGM_REG_COUNTER(&pStats->StatR3ChunkR3MapTlbMisses, "/PGM/ChunkR3Map/TlbMissesR3", "TLB misses.");
1693	PGM_REG_COUNTER(&pStats->StatR3PageMapTlbHits, "/PGM/R3/Page/MapTlbHits", "TLB hits.");
1694	PGM_REG_COUNTER(&pStats->StatR3PageMapTlbMisses, "/PGM/R3/Page/MapTlbMisses", "TLB misses.");
1695	PGM_REG_COUNTER(&pStats->StatPageMapTlbFlushes, "/PGM/R3/Page/MapTlbFlushes", "TLB flushes (all contexts).");
1696	PGM_REG_COUNTER(&pStats->StatPageMapTlbFlushEntry, "/PGM/R3/Page/MapTlbFlushEntry", "TLB entry flushes (all contexts).");
1697
1698	PGM_REG_COUNTER(&pStats->StatRZRamRangeTlbHits, "/PGM/RZ/RamRange/TlbHits", "TLB hits.");
1699	PGM_REG_COUNTER(&pStats->StatRZRamRangeTlbMisses, "/PGM/RZ/RamRange/TlbMisses", "TLB misses.");
1700	PGM_REG_COUNTER(&pStats->StatR3RamRangeTlbHits, "/PGM/R3/RamRange/TlbHits", "TLB hits.");
1701	PGM_REG_COUNTER(&pStats->StatR3RamRangeTlbMisses, "/PGM/R3/RamRange/TlbMisses", "TLB misses.");
1702
1703	PGM_REG_PROFILE(&pStats->StatRZSyncCR3HandlerVirtualUpdate, "/PGM/RZ/SyncCR3/Handlers/VirtualUpdate", "Profiling of the virtual handler updates.");
1704	PGM_REG_PROFILE(&pStats->StatRZSyncCR3HandlerVirtualReset, "/PGM/RZ/SyncCR3/Handlers/VirtualReset", "Profiling of the virtual handler resets.");
1705	PGM_REG_PROFILE(&pStats->StatR3SyncCR3HandlerVirtualUpdate, "/PGM/R3/SyncCR3/Handlers/VirtualUpdate", "Profiling of the virtual handler updates.");
1706	PGM_REG_PROFILE(&pStats->StatR3SyncCR3HandlerVirtualReset, "/PGM/R3/SyncCR3/Handlers/VirtualReset", "Profiling of the virtual handler resets.");
1707
1708	PGM_REG_COUNTER(&pStats->StatRZPhysHandlerReset, "/PGM/RZ/PhysHandlerReset", "The number of times PGMHandlerPhysicalReset is called.");
1709	PGM_REG_COUNTER(&pStats->StatR3PhysHandlerReset, "/PGM/R3/PhysHandlerReset", "The number of times PGMHandlerPhysicalReset is called.");
1710	PGM_REG_COUNTER(&pStats->StatRZPhysHandlerLookupHits, "/PGM/RZ/PhysHandlerLookupHits", "The number of cache hits when looking up physical handlers.");
1711	PGM_REG_COUNTER(&pStats->StatR3PhysHandlerLookupHits, "/PGM/R3/PhysHandlerLookupHits", "The number of cache hits when looking up physical handlers.");
1712	PGM_REG_COUNTER(&pStats->StatRZPhysHandlerLookupMisses, "/PGM/RZ/PhysHandlerLookupMisses", "The number of cache misses when looking up physical handlers.");
1713	PGM_REG_COUNTER(&pStats->StatR3PhysHandlerLookupMisses, "/PGM/R3/PhysHandlerLookupMisses", "The number of cache misses when looking up physical handlers.");
1714	PGM_REG_PROFILE(&pStats->StatRZVirtHandlerSearchByPhys, "/PGM/RZ/VirtHandlerSearchByPhys", "Profiling of pgmHandlerVirtualFindByPhysAddr.");
1715	PGM_REG_PROFILE(&pStats->StatR3VirtHandlerSearchByPhys, "/PGM/R3/VirtHandlerSearchByPhys", "Profiling of pgmHandlerVirtualFindByPhysAddr.");
1716
1717	PGM_REG_COUNTER(&pStats->StatRZPageReplaceShared, "/PGM/RZ/Page/ReplacedShared", "Times a shared page was replaced.");
1718	PGM_REG_COUNTER(&pStats->StatRZPageReplaceZero, "/PGM/RZ/Page/ReplacedZero", "Times the zero page was replaced.");
1719	/// @todo PGM_REG_COUNTER(&pStats->StatRZPageHandyAllocs, "/PGM/RZ/Page/HandyAllocs", "Number of times we've allocated more handy pages.");
1720	PGM_REG_COUNTER(&pStats->StatR3PageReplaceShared, "/PGM/R3/Page/ReplacedShared", "Times a shared page was replaced.");
1721	PGM_REG_COUNTER(&pStats->StatR3PageReplaceZero, "/PGM/R3/Page/ReplacedZero", "Times the zero page was replaced.");
1722	/// @todo PGM_REG_COUNTER(&pStats->StatR3PageHandyAllocs, "/PGM/R3/Page/HandyAllocs", "Number of times we've allocated more handy pages.");
1723
1724	PGM_REG_COUNTER(&pStats->StatRZPhysRead, "/PGM/RZ/Phys/Read", "The number of times PGMPhysRead was called.");
1725	PGM_REG_COUNTER_BYTES(&pStats->StatRZPhysReadBytes, "/PGM/RZ/Phys/Read/Bytes", "The number of bytes read by PGMPhysRead.");
1726	PGM_REG_COUNTER(&pStats->StatRZPhysWrite, "/PGM/RZ/Phys/Write", "The number of times PGMPhysWrite was called.");
1727	PGM_REG_COUNTER_BYTES(&pStats->StatRZPhysWriteBytes, "/PGM/RZ/Phys/Write/Bytes", "The number of bytes written by PGMPhysWrite.");
1728	PGM_REG_COUNTER(&pStats->StatRZPhysSimpleRead, "/PGM/RZ/Phys/Simple/Read", "The number of times PGMPhysSimpleReadGCPtr was called.");
1729	PGM_REG_COUNTER_BYTES(&pStats->StatRZPhysSimpleReadBytes, "/PGM/RZ/Phys/Simple/Read/Bytes", "The number of bytes read by PGMPhysSimpleReadGCPtr.");
1730	PGM_REG_COUNTER(&pStats->StatRZPhysSimpleWrite, "/PGM/RZ/Phys/Simple/Write", "The number of times PGMPhysSimpleWriteGCPtr was called.");
1731	PGM_REG_COUNTER_BYTES(&pStats->StatRZPhysSimpleWriteBytes, "/PGM/RZ/Phys/Simple/Write/Bytes", "The number of bytes written by PGMPhysSimpleWriteGCPtr.");
1732
1733	/* GC only: */
1734	PGM_REG_COUNTER(&pStats->StatRCInvlPgConflict, "/PGM/RC/InvlPgConflict", "Number of times PGMInvalidatePage() detected a mapping conflict.");
1735	PGM_REG_COUNTER(&pStats->StatRCInvlPgSyncMonCR3, "/PGM/RC/InvlPgSyncMonitorCR3", "Number of times PGMInvalidatePage() ran into PGM_SYNC_MONITOR_CR3.");
1736
1737	PGM_REG_COUNTER(&pStats->StatRCPhysRead, "/PGM/RC/Phys/Read", "The number of times PGMPhysRead was called.");
1738	PGM_REG_COUNTER_BYTES(&pStats->StatRCPhysReadBytes, "/PGM/RC/Phys/Read/Bytes", "The number of bytes read by PGMPhysRead.");
1739	PGM_REG_COUNTER(&pStats->StatRCPhysWrite, "/PGM/RC/Phys/Write", "The number of times PGMPhysWrite was called.");
1740	PGM_REG_COUNTER_BYTES(&pStats->StatRCPhysWriteBytes, "/PGM/RC/Phys/Write/Bytes", "The number of bytes written by PGMPhysWrite.");
1741	PGM_REG_COUNTER(&pStats->StatRCPhysSimpleRead, "/PGM/RC/Phys/Simple/Read", "The number of times PGMPhysSimpleReadGCPtr was called.");
1742	PGM_REG_COUNTER_BYTES(&pStats->StatRCPhysSimpleReadBytes, "/PGM/RC/Phys/Simple/Read/Bytes", "The number of bytes read by PGMPhysSimpleReadGCPtr.");
1743	PGM_REG_COUNTER(&pStats->StatRCPhysSimpleWrite, "/PGM/RC/Phys/Simple/Write", "The number of times PGMPhysSimpleWriteGCPtr was called.");
1744	PGM_REG_COUNTER_BYTES(&pStats->StatRCPhysSimpleWriteBytes, "/PGM/RC/Phys/Simple/Write/Bytes", "The number of bytes written by PGMPhysSimpleWriteGCPtr.");
1745
1746	PGM_REG_COUNTER(&pStats->StatTrackVirgin, "/PGM/Track/Virgin", "The number of first time shadowings");
1747	PGM_REG_COUNTER(&pStats->StatTrackAliased, "/PGM/Track/Aliased", "The number of times switching to cRef2, i.e. the page is being shadowed by two PTs.");
1748	PGM_REG_COUNTER(&pStats->StatTrackAliasedMany, "/PGM/Track/AliasedMany", "The number of times we're tracking using cRef2.");
1749	PGM_REG_COUNTER(&pStats->StatTrackAliasedLots, "/PGM/Track/AliasedLots", "The number of times we're hitting pages which has overflowed cRef2");
1750	PGM_REG_COUNTER(&pStats->StatTrackOverflows, "/PGM/Track/Overflows", "The number of times the extent list grows too long.");
1751	PGM_REG_COUNTER(&pStats->StatTrackNoExtentsLeft, "/PGM/Track/NoExtentLeft", "The number of times the extent list was exhausted.");
1752	PGM_REG_PROFILE(&pStats->StatTrackDeref, "/PGM/Track/Deref", "Profiling of SyncPageWorkerTrackDeref (expensive).");
1753
1754	# undef PGM_REG_COUNTER
1755	# undef PGM_REG_PROFILE
1756	#endif
1757
1758	/*
1759	* Note! The layout below matches the member layout exactly!
1760	*/
1761
1762	/*
1763	* Common - stats
1764	*/
1765	for (VMCPUID idCpu = 0; idCpu < pVM->cCpus; idCpu++)
1766	{
1767	PPGMCPU pPgmCpu = &pVM->aCpus[idCpu].pgm.s;
1768
1769	#define PGM_REG_COUNTER(a, b, c) \
1770	rc = STAMR3RegisterF(pVM, a, STAMTYPE_COUNTER, STAMVISIBILITY_ALWAYS, STAMUNIT_OCCURENCES, c, b, idCpu); \
1771	AssertRC(rc);
1772	#define PGM_REG_PROFILE(a, b, c) \
1773	rc = STAMR3RegisterF(pVM, a, STAMTYPE_PROFILE, STAMVISIBILITY_ALWAYS, STAMUNIT_TICKS_PER_CALL, c, b, idCpu); \
1774	AssertRC(rc);
1775
1776	PGM_REG_COUNTER(&pPgmCpu->cGuestModeChanges, "/PGM/CPU%u/cGuestModeChanges", "Number of guest mode changes.");
1777
1778	#ifdef VBOX_WITH_STATISTICS
1779	PGMCPUSTATS *pCpuStats = pVM->aCpus[idCpu].pgm.s.pStatsR3;
1780
1781	# if 0 /* rarely useful; leave for debugging. */
1782	for (unsigned j = 0; j < RT_ELEMENTS(pPgmCpu->StatSyncPtPD); j++)
1783	STAMR3RegisterF(pVM, &pCpuStats->StatSyncPtPD[i], STAMTYPE_COUNTER, STAMVISIBILITY_USED, STAMUNIT_OCCURENCES,
1784	"The number of SyncPT per PD n.", "/PGM/CPU%u/PDSyncPT/%04X", i, j);
1785	for (unsigned j = 0; j < RT_ELEMENTS(pCpuStats->StatSyncPagePD); j++)
1786	STAMR3RegisterF(pVM, &pCpuStats->StatSyncPagePD[i], STAMTYPE_COUNTER, STAMVISIBILITY_USED, STAMUNIT_OCCURENCES,
1787	"The number of SyncPage per PD n.", "/PGM/CPU%u/PDSyncPage/%04X", i, j);
1788	# endif
1789	/* R0 only: */
1790	PGM_REG_PROFILE(&pCpuStats->StatR0NpMiscfg, "/PGM/CPU%u/R0/NpMiscfg", "PGMR0Trap0eHandlerNPMisconfig() profiling.");
1791	PGM_REG_COUNTER(&pCpuStats->StatR0NpMiscfgSyncPage, "/PGM/CPU%u/R0/NpMiscfgSyncPage", "SyncPage calls from PGMR0Trap0eHandlerNPMisconfig().");
1792
1793	/* RZ only: */
1794	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0e, "/PGM/CPU%u/RZ/Trap0e", "Profiling of the PGMTrap0eHandler() body.");
1795	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2Ballooned, "/PGM/CPU%u/RZ/Trap0e/Time2/Ballooned", "Profiling of the Trap0eHandler body when the cause is read access to a ballooned page.");
1796	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2CSAM, "/PGM/CPU%u/RZ/Trap0e/Time2/CSAM", "Profiling of the Trap0eHandler body when the cause is CSAM.");
1797	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2DirtyAndAccessed, "/PGM/CPU%u/RZ/Trap0e/Time2/DirtyAndAccessedBits", "Profiling of the Trap0eHandler body when the cause is dirty and/or accessed bit emulation.");
1798	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2GuestTrap, "/PGM/CPU%u/RZ/Trap0e/Time2/GuestTrap", "Profiling of the Trap0eHandler body when the cause is a guest trap.");
1799	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2HndPhys, "/PGM/CPU%u/RZ/Trap0e/Time2/HandlerPhysical", "Profiling of the Trap0eHandler body when the cause is a physical handler.");
1800	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2HndVirt, "/PGM/CPU%u/RZ/Trap0e/Time2/HandlerVirtual", "Profiling of the Trap0eHandler body when the cause is a virtual handler.");
1801	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2HndUnhandled, "/PGM/CPU%u/RZ/Trap0e/Time2/HandlerUnhandled", "Profiling of the Trap0eHandler body when the cause is access outside the monitored areas of a monitored page.");
1802	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2InvalidPhys, "/PGM/CPU%u/RZ/Trap0e/Time2/InvalidPhys", "Profiling of the Trap0eHandler body when the cause is access to an invalid physical guest address.");
1803	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2MakeWritable, "/PGM/CPU%u/RZ/Trap0e/Time2/MakeWritable", "Profiling of the Trap0eHandler body when the cause is that a page needed to be made writeable.");
1804	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2Mapping, "/PGM/CPU%u/RZ/Trap0e/Time2/Mapping", "Profiling of the Trap0eHandler body when the cause is related to the guest mappings.");
1805	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2Misc, "/PGM/CPU%u/RZ/Trap0e/Time2/Misc", "Profiling of the Trap0eHandler body when the cause is not known.");
1806	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2OutOfSync, "/PGM/CPU%u/RZ/Trap0e/Time2/OutOfSync", "Profiling of the Trap0eHandler body when the cause is an out-of-sync page.");
1807	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2OutOfSyncHndPhys, "/PGM/CPU%u/RZ/Trap0e/Time2/OutOfSyncHndPhys", "Profiling of the Trap0eHandler body when the cause is an out-of-sync physical handler page.");
1808	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2OutOfSyncHndVirt, "/PGM/CPU%u/RZ/Trap0e/Time2/OutOfSyncHndVirt", "Profiling of the Trap0eHandler body when the cause is an out-of-sync virtual handler page.");
1809	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2OutOfSyncHndObs, "/PGM/CPU%u/RZ/Trap0e/Time2/OutOfSyncObsHnd", "Profiling of the Trap0eHandler body when the cause is an obsolete handler page.");
1810	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2SyncPT, "/PGM/CPU%u/RZ/Trap0e/Time2/SyncPT", "Profiling of the Trap0eHandler body when the cause is lazy syncing of a PT.");
1811	PGM_REG_PROFILE(&pCpuStats->StatRZTrap0eTime2WPEmulation, "/PGM/CPU%u/RZ/Trap0e/Time2/WPEmulation", "Profiling of the Trap0eHandler body when the cause is CR0.WP emulation.");
1812	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eConflicts, "/PGM/CPU%u/RZ/Trap0e/Conflicts", "The number of times #PF was caused by an undetected conflict.");
1813	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersMapping, "/PGM/CPU%u/RZ/Trap0e/Handlers/Mapping", "Number of traps due to access handlers in mappings.");
1814	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersOutOfSync, "/PGM/CPU%u/RZ/Trap0e/Handlers/OutOfSync", "Number of traps due to out-of-sync handled pages.");
1815	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersPhysAll, "/PGM/CPU%u/RZ/Trap0e/Handlers/PhysAll", "Number of traps due to physical all-access handlers.");
1816	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersPhysAllOpt, "/PGM/CPU%u/RZ/Trap0e/Handlers/PhysAllOpt", "Number of the physical all-access handler traps using the optimization.");
1817	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersPhysWrite, "/PGM/CPU%u/RZ/Trap0e/Handlers/PhysWrite", "Number of traps due to physical write-access handlers.");
1818	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersVirtual, "/PGM/CPU%u/RZ/Trap0e/Handlers/Virtual", "Number of traps due to virtual access handlers.");
1819	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersVirtualByPhys, "/PGM/CPU%u/RZ/Trap0e/Handlers/VirtualByPhys", "Number of traps due to virtual access handlers by physical address.");
1820	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersVirtualUnmarked,"/PGM/CPU%u/RZ/Trap0e/Handlers/VirtualUnmarked","Number of traps due to virtual access handlers by virtual address (without proper physical flags).");
1821	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersUnhandled, "/PGM/CPU%u/RZ/Trap0e/Handlers/Unhandled", "Number of traps due to access outside range of monitored page(s).");
1822	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eHandlersInvalid, "/PGM/CPU%u/RZ/Trap0e/Handlers/Invalid", "Number of traps due to access to invalid physical memory.");
1823	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eUSNotPresentRead, "/PGM/CPU%u/RZ/Trap0e/Err/User/NPRead", "Number of user mode not present read page faults.");
1824	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eUSNotPresentWrite, "/PGM/CPU%u/RZ/Trap0e/Err/User/NPWrite", "Number of user mode not present write page faults.");
1825	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eUSWrite, "/PGM/CPU%u/RZ/Trap0e/Err/User/Write", "Number of user mode write page faults.");
1826	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eUSReserved, "/PGM/CPU%u/RZ/Trap0e/Err/User/Reserved", "Number of user mode reserved bit page faults.");
1827	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eUSNXE, "/PGM/CPU%u/RZ/Trap0e/Err/User/NXE", "Number of user mode NXE page faults.");
1828	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eUSRead, "/PGM/CPU%u/RZ/Trap0e/Err/User/Read", "Number of user mode read page faults.");
1829	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eSVNotPresentRead, "/PGM/CPU%u/RZ/Trap0e/Err/Supervisor/NPRead", "Number of supervisor mode not present read page faults.");
1830	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eSVNotPresentWrite, "/PGM/CPU%u/RZ/Trap0e/Err/Supervisor/NPWrite", "Number of supervisor mode not present write page faults.");
1831	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eSVWrite, "/PGM/CPU%u/RZ/Trap0e/Err/Supervisor/Write", "Number of supervisor mode write page faults.");
1832	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eSVReserved, "/PGM/CPU%u/RZ/Trap0e/Err/Supervisor/Reserved", "Number of supervisor mode reserved bit page faults.");
1833	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eSNXE, "/PGM/CPU%u/RZ/Trap0e/Err/Supervisor/NXE", "Number of supervisor mode NXE page faults.");
1834	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eGuestPF, "/PGM/CPU%u/RZ/Trap0e/GuestPF", "Number of real guest page faults.");
1835	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eGuestPFMapping, "/PGM/CPU%u/RZ/Trap0e/GuestPF/InMapping", "Number of real guest page faults in a mapping.");
1836	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eWPEmulInRZ, "/PGM/CPU%u/RZ/Trap0e/WP/InRZ", "Number of guest page faults due to X86_CR0_WP emulation.");
1837	PGM_REG_COUNTER(&pCpuStats->StatRZTrap0eWPEmulToR3, "/PGM/CPU%u/RZ/Trap0e/WP/ToR3", "Number of guest page faults due to X86_CR0_WP emulation (forward to R3 for emulation).");
1838	#if 0 /* rarely useful; leave for debugging. */
1839	for (unsigned j = 0; j < RT_ELEMENTS(pCpuStats->StatRZTrap0ePD); j++)
1840	STAMR3RegisterF(pVM, &pCpuStats->StatRZTrap0ePD[i], STAMTYPE_COUNTER, STAMVISIBILITY_USED, STAMUNIT_OCCURENCES,
1841	"The number of traps in page directory n.", "/PGM/CPU%u/RZ/Trap0e/PD/%04X", i, j);
1842	#endif
1843	PGM_REG_COUNTER(&pCpuStats->StatRZGuestCR3WriteHandled, "/PGM/CPU%u/RZ/CR3WriteHandled", "The number of times the Guest CR3 change was successfully handled.");
1844	PGM_REG_COUNTER(&pCpuStats->StatRZGuestCR3WriteUnhandled, "/PGM/CPU%u/RZ/CR3WriteUnhandled", "The number of times the Guest CR3 change was passed back to the recompiler.");
1845	PGM_REG_COUNTER(&pCpuStats->StatRZGuestCR3WriteConflict, "/PGM/CPU%u/RZ/CR3WriteConflict", "The number of times the Guest CR3 monitoring detected a conflict.");
1846	PGM_REG_COUNTER(&pCpuStats->StatRZGuestROMWriteHandled, "/PGM/CPU%u/RZ/ROMWriteHandled", "The number of times the Guest ROM change was successfully handled.");
1847	PGM_REG_COUNTER(&pCpuStats->StatRZGuestROMWriteUnhandled, "/PGM/CPU%u/RZ/ROMWriteUnhandled", "The number of times the Guest ROM change was passed back to the recompiler.");
1848
1849	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapMigrateInvlPg, "/PGM/CPU%u/RZ/DynMap/MigrateInvlPg", "invlpg count in PGMR0DynMapMigrateAutoSet.");
1850	PGM_REG_PROFILE(&pCpuStats->StatRZDynMapGCPageInl, "/PGM/CPU%u/RZ/DynMap/PageGCPageInl", "Calls to pgmR0DynMapGCPageInlined.");
1851	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapGCPageInlHits, "/PGM/CPU%u/RZ/DynMap/PageGCPageInl/Hits", "Hash table lookup hits.");
1852	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapGCPageInlMisses, "/PGM/CPU%u/RZ/DynMap/PageGCPageInl/Misses", "Misses that falls back to the code common.");
1853	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapGCPageInlRamHits, "/PGM/CPU%u/RZ/DynMap/PageGCPageInl/RamHits", "1st ram range hits.");
1854	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapGCPageInlRamMisses, "/PGM/CPU%u/RZ/DynMap/PageGCPageInl/RamMisses", "1st ram range misses, takes slow path.");
1855	PGM_REG_PROFILE(&pCpuStats->StatRZDynMapHCPageInl, "/PGM/CPU%u/RZ/DynMap/PageHCPageInl", "Calls to pgmRZDynMapHCPageInlined.");
1856	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapHCPageInlHits, "/PGM/CPU%u/RZ/DynMap/PageHCPageInl/Hits", "Hash table lookup hits.");
1857	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapHCPageInlMisses, "/PGM/CPU%u/RZ/DynMap/PageHCPageInl/Misses", "Misses that falls back to the code common.");
1858	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPage, "/PGM/CPU%u/RZ/DynMap/Page", "Calls to pgmR0DynMapPage");
1859	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapSetOptimize, "/PGM/CPU%u/RZ/DynMap/Page/SetOptimize", "Calls to pgmRZDynMapOptimizeAutoSet.");
1860	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapSetSearchFlushes, "/PGM/CPU%u/RZ/DynMap/Page/SetSearchFlushes", "Set search restoring to subset flushes.");
1861	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapSetSearchHits, "/PGM/CPU%u/RZ/DynMap/Page/SetSearchHits", "Set search hits.");
1862	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapSetSearchMisses, "/PGM/CPU%u/RZ/DynMap/Page/SetSearchMisses", "Set search misses.");
1863	PGM_REG_PROFILE(&pCpuStats->StatRZDynMapHCPage, "/PGM/CPU%u/RZ/DynMap/Page/HCPage", "Calls to pgmRZDynMapHCPageCommon (ring-0).");
1864	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageHits0, "/PGM/CPU%u/RZ/DynMap/Page/Hits0", "Hits at iPage+0");
1865	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageHits1, "/PGM/CPU%u/RZ/DynMap/Page/Hits1", "Hits at iPage+1");
1866	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageHits2, "/PGM/CPU%u/RZ/DynMap/Page/Hits2", "Hits at iPage+2");
1867	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageInvlPg, "/PGM/CPU%u/RZ/DynMap/Page/InvlPg", "invlpg count in pgmR0DynMapPageSlow.");
1868	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageSlow, "/PGM/CPU%u/RZ/DynMap/Page/Slow", "Calls to pgmR0DynMapPageSlow - subtract this from pgmR0DynMapPage to get 1st level hits.");
1869	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageSlowLoopHits, "/PGM/CPU%u/RZ/DynMap/Page/SlowLoopHits" , "Hits in the loop path.");
1870	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageSlowLoopMisses, "/PGM/CPU%u/RZ/DynMap/Page/SlowLoopMisses", "Misses in the loop path. NonLoopMisses = Slow - SlowLoopHit - SlowLoopMisses");
1871	//PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPageSlowLostHits, "/PGM/CPU%u/R0/DynMap/Page/SlowLostHits", "Lost hits.");
1872	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapSubsets, "/PGM/CPU%u/RZ/DynMap/Subsets", "Times PGMRZDynMapPushAutoSubset was called.");
1873	PGM_REG_COUNTER(&pCpuStats->StatRZDynMapPopFlushes, "/PGM/CPU%u/RZ/DynMap/SubsetPopFlushes", "Times PGMRZDynMapPopAutoSubset flushes the subset.");
1874	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[0], "/PGM/CPU%u/RZ/DynMap/SetFilledPct000..09", "00-09% filled (RC: min(set-size, dynmap-size))");
1875	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[1], "/PGM/CPU%u/RZ/DynMap/SetFilledPct010..19", "10-19% filled (RC: min(set-size, dynmap-size))");
1876	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[2], "/PGM/CPU%u/RZ/DynMap/SetFilledPct020..29", "20-29% filled (RC: min(set-size, dynmap-size))");
1877	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[3], "/PGM/CPU%u/RZ/DynMap/SetFilledPct030..39", "30-39% filled (RC: min(set-size, dynmap-size))");
1878	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[4], "/PGM/CPU%u/RZ/DynMap/SetFilledPct040..49", "40-49% filled (RC: min(set-size, dynmap-size))");
1879	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[5], "/PGM/CPU%u/RZ/DynMap/SetFilledPct050..59", "50-59% filled (RC: min(set-size, dynmap-size))");
1880	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[6], "/PGM/CPU%u/RZ/DynMap/SetFilledPct060..69", "60-69% filled (RC: min(set-size, dynmap-size))");
1881	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[7], "/PGM/CPU%u/RZ/DynMap/SetFilledPct070..79", "70-79% filled (RC: min(set-size, dynmap-size))");
1882	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[8], "/PGM/CPU%u/RZ/DynMap/SetFilledPct080..89", "80-89% filled (RC: min(set-size, dynmap-size))");
1883	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[9], "/PGM/CPU%u/RZ/DynMap/SetFilledPct090..99", "90-99% filled (RC: min(set-size, dynmap-size))");
1884	PGM_REG_COUNTER(&pCpuStats->aStatRZDynMapSetFilledPct[10], "/PGM/CPU%u/RZ/DynMap/SetFilledPct100", "100% filled (RC: min(set-size, dynmap-size))");
1885
1886	/* HC only: */
1887
1888	/* RZ & R3: */
1889	PGM_REG_PROFILE(&pCpuStats->StatRZSyncCR3, "/PGM/CPU%u/RZ/SyncCR3", "Profiling of the PGMSyncCR3() body.");
1890	PGM_REG_PROFILE(&pCpuStats->StatRZSyncCR3Handlers, "/PGM/CPU%u/RZ/SyncCR3/Handlers", "Profiling of the PGMSyncCR3() update handler section.");
1891	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3Global, "/PGM/CPU%u/RZ/SyncCR3/Global", "The number of global CR3 syncs.");
1892	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3NotGlobal, "/PGM/CPU%u/RZ/SyncCR3/NotGlobal", "The number of non-global CR3 syncs.");
1893	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3DstCacheHit, "/PGM/CPU%u/RZ/SyncCR3/DstChacheHit", "The number of times we got some kind of a cache hit.");
1894	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3DstFreed, "/PGM/CPU%u/RZ/SyncCR3/DstFreed", "The number of times we've had to free a shadow entry.");
1895	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3DstFreedSrcNP, "/PGM/CPU%u/RZ/SyncCR3/DstFreedSrcNP", "The number of times we've had to free a shadow entry for which the source entry was not present.");
1896	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3DstNotPresent, "/PGM/CPU%u/RZ/SyncCR3/DstNotPresent", "The number of times we've encountered a not present shadow entry for a present guest entry.");
1897	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3DstSkippedGlobalPD, "/PGM/CPU%u/RZ/SyncCR3/DstSkippedGlobalPD", "The number of times a global page directory wasn't flushed.");
1898	PGM_REG_COUNTER(&pCpuStats->StatRZSyncCR3DstSkippedGlobalPT, "/PGM/CPU%u/RZ/SyncCR3/DstSkippedGlobalPT", "The number of times a page table with only global entries wasn't flushed.");
1899	PGM_REG_PROFILE(&pCpuStats->StatRZSyncPT, "/PGM/CPU%u/RZ/SyncPT", "Profiling of the pfnSyncPT() body.");
1900	PGM_REG_COUNTER(&pCpuStats->StatRZSyncPTFailed, "/PGM/CPU%u/RZ/SyncPT/Failed", "The number of times pfnSyncPT() failed.");
1901	PGM_REG_COUNTER(&pCpuStats->StatRZSyncPT4K, "/PGM/CPU%u/RZ/SyncPT/4K", "Nr of 4K PT syncs");
1902	PGM_REG_COUNTER(&pCpuStats->StatRZSyncPT4M, "/PGM/CPU%u/RZ/SyncPT/4M", "Nr of 4M PT syncs");
1903	PGM_REG_COUNTER(&pCpuStats->StatRZSyncPagePDNAs, "/PGM/CPU%u/RZ/SyncPagePDNAs", "The number of time we've marked a PD not present from SyncPage to virtualize the accessed bit.");
1904	PGM_REG_COUNTER(&pCpuStats->StatRZSyncPagePDOutOfSync, "/PGM/CPU%u/RZ/SyncPagePDOutOfSync", "The number of time we've encountered an out-of-sync PD in SyncPage.");
1905	PGM_REG_COUNTER(&pCpuStats->StatRZAccessedPage, "/PGM/CPU%u/RZ/AccessedPage", "The number of pages marked not present for accessed bit emulation.");
1906	PGM_REG_PROFILE(&pCpuStats->StatRZDirtyBitTracking, "/PGM/CPU%u/RZ/DirtyPage", "Profiling the dirty bit tracking in CheckPageFault().");
1907	PGM_REG_COUNTER(&pCpuStats->StatRZDirtyPage, "/PGM/CPU%u/RZ/DirtyPage/Mark", "The number of pages marked read-only for dirty bit tracking.");
1908	PGM_REG_COUNTER(&pCpuStats->StatRZDirtyPageBig, "/PGM/CPU%u/RZ/DirtyPage/MarkBig", "The number of 4MB pages marked read-only for dirty bit tracking.");
1909	PGM_REG_COUNTER(&pCpuStats->StatRZDirtyPageSkipped, "/PGM/CPU%u/RZ/DirtyPage/Skipped", "The number of pages already dirty or readonly.");
1910	PGM_REG_COUNTER(&pCpuStats->StatRZDirtyPageTrap, "/PGM/CPU%u/RZ/DirtyPage/Trap", "The number of traps generated for dirty bit tracking.");
1911	PGM_REG_COUNTER(&pCpuStats->StatRZDirtyPageStale, "/PGM/CPU%u/RZ/DirtyPage/Stale", "The number of traps generated for dirty bit tracking (stale tlb entries).");
1912	PGM_REG_COUNTER(&pCpuStats->StatRZDirtiedPage, "/PGM/CPU%u/RZ/DirtyPage/SetDirty", "The number of pages marked dirty because of write accesses.");
1913	PGM_REG_COUNTER(&pCpuStats->StatRZDirtyTrackRealPF, "/PGM/CPU%u/RZ/DirtyPage/RealPF", "The number of real pages faults during dirty bit tracking.");
1914	PGM_REG_COUNTER(&pCpuStats->StatRZPageAlreadyDirty, "/PGM/CPU%u/RZ/DirtyPage/AlreadySet", "The number of pages already marked dirty because of write accesses.");
1915	PGM_REG_PROFILE(&pCpuStats->StatRZInvalidatePage, "/PGM/CPU%u/RZ/InvalidatePage", "PGMInvalidatePage() profiling.");
1916	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePage4KBPages, "/PGM/CPU%u/RZ/InvalidatePage/4KBPages", "The number of times PGMInvalidatePage() was called for a 4KB page.");
1917	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePage4MBPages, "/PGM/CPU%u/RZ/InvalidatePage/4MBPages", "The number of times PGMInvalidatePage() was called for a 4MB page.");
1918	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePage4MBPagesSkip, "/PGM/CPU%u/RZ/InvalidatePage/4MBPagesSkip","The number of times PGMInvalidatePage() skipped a 4MB page.");
1919	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePagePDMappings, "/PGM/CPU%u/RZ/InvalidatePage/PDMappings", "The number of times PGMInvalidatePage() was called for a page directory containing mappings (no conflict).");
1920	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePagePDNAs, "/PGM/CPU%u/RZ/InvalidatePage/PDNAs", "The number of times PGMInvalidatePage() was called for a not accessed page directory.");
1921	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePagePDNPs, "/PGM/CPU%u/RZ/InvalidatePage/PDNPs", "The number of times PGMInvalidatePage() was called for a not present page directory.");
1922	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePagePDOutOfSync, "/PGM/CPU%u/RZ/InvalidatePage/PDOutOfSync", "The number of times PGMInvalidatePage() was called for an out of sync page directory.");
1923	PGM_REG_COUNTER(&pCpuStats->StatRZInvalidatePageSkipped, "/PGM/CPU%u/RZ/InvalidatePage/Skipped", "The number of times PGMInvalidatePage() was skipped due to not present shw or pending pending SyncCR3.");
1924	PGM_REG_COUNTER(&pCpuStats->StatRZPageOutOfSyncSupervisor, "/PGM/CPU%u/RZ/OutOfSync/SuperVisor", "Number of traps due to pages out of sync (P) and times VerifyAccessSyncPage calls SyncPage.");
1925	PGM_REG_COUNTER(&pCpuStats->StatRZPageOutOfSyncUser, "/PGM/CPU%u/RZ/OutOfSync/User", "Number of traps due to pages out of sync (P) and times VerifyAccessSyncPage calls SyncPage.");
1926	PGM_REG_COUNTER(&pCpuStats->StatRZPageOutOfSyncSupervisorWrite,"/PGM/CPU%u/RZ/OutOfSync/SuperVisorWrite", "Number of traps due to pages out of sync (RW) and times VerifyAccessSyncPage calls SyncPage.");
1927	PGM_REG_COUNTER(&pCpuStats->StatRZPageOutOfSyncUserWrite, "/PGM/CPU%u/RZ/OutOfSync/UserWrite", "Number of traps due to pages out of sync (RW) and times VerifyAccessSyncPage calls SyncPage.");
1928	PGM_REG_COUNTER(&pCpuStats->StatRZPageOutOfSyncBallloon, "/PGM/CPU%u/RZ/OutOfSync/Balloon", "The number of times a ballooned page was accessed (read).");
1929	PGM_REG_PROFILE(&pCpuStats->StatRZPrefetch, "/PGM/CPU%u/RZ/Prefetch", "PGMPrefetchPage profiling.");
1930	PGM_REG_PROFILE(&pCpuStats->StatRZFlushTLB, "/PGM/CPU%u/RZ/FlushTLB", "Profiling of the PGMFlushTLB() body.");
1931	PGM_REG_COUNTER(&pCpuStats->StatRZFlushTLBNewCR3, "/PGM/CPU%u/RZ/FlushTLB/NewCR3", "The number of times PGMFlushTLB was called with a new CR3, non-global. (switch)");
1932	PGM_REG_COUNTER(&pCpuStats->StatRZFlushTLBNewCR3Global, "/PGM/CPU%u/RZ/FlushTLB/NewCR3Global", "The number of times PGMFlushTLB was called with a new CR3, global. (switch)");
1933	PGM_REG_COUNTER(&pCpuStats->StatRZFlushTLBSameCR3, "/PGM/CPU%u/RZ/FlushTLB/SameCR3", "The number of times PGMFlushTLB was called with the same CR3, non-global. (flush)");
1934	PGM_REG_COUNTER(&pCpuStats->StatRZFlushTLBSameCR3Global, "/PGM/CPU%u/RZ/FlushTLB/SameCR3Global", "The number of times PGMFlushTLB was called with the same CR3, global. (flush)");
1935	PGM_REG_PROFILE(&pCpuStats->StatRZGstModifyPage, "/PGM/CPU%u/RZ/GstModifyPage", "Profiling of the PGMGstModifyPage() body.");
1936
1937	PGM_REG_PROFILE(&pCpuStats->StatR3SyncCR3, "/PGM/CPU%u/R3/SyncCR3", "Profiling of the PGMSyncCR3() body.");
1938	PGM_REG_PROFILE(&pCpuStats->StatR3SyncCR3Handlers, "/PGM/CPU%u/R3/SyncCR3/Handlers", "Profiling of the PGMSyncCR3() update handler section.");
1939	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3Global, "/PGM/CPU%u/R3/SyncCR3/Global", "The number of global CR3 syncs.");
1940	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3NotGlobal, "/PGM/CPU%u/R3/SyncCR3/NotGlobal", "The number of non-global CR3 syncs.");
1941	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3DstCacheHit, "/PGM/CPU%u/R3/SyncCR3/DstChacheHit", "The number of times we got some kind of a cache hit.");
1942	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3DstFreed, "/PGM/CPU%u/R3/SyncCR3/DstFreed", "The number of times we've had to free a shadow entry.");
1943	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3DstFreedSrcNP, "/PGM/CPU%u/R3/SyncCR3/DstFreedSrcNP", "The number of times we've had to free a shadow entry for which the source entry was not present.");
1944	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3DstNotPresent, "/PGM/CPU%u/R3/SyncCR3/DstNotPresent", "The number of times we've encountered a not present shadow entry for a present guest entry.");
1945	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3DstSkippedGlobalPD, "/PGM/CPU%u/R3/SyncCR3/DstSkippedGlobalPD", "The number of times a global page directory wasn't flushed.");
1946	PGM_REG_COUNTER(&pCpuStats->StatR3SyncCR3DstSkippedGlobalPT, "/PGM/CPU%u/R3/SyncCR3/DstSkippedGlobalPT", "The number of times a page table with only global entries wasn't flushed.");
1947	PGM_REG_PROFILE(&pCpuStats->StatR3SyncPT, "/PGM/CPU%u/R3/SyncPT", "Profiling of the pfnSyncPT() body.");
1948	PGM_REG_COUNTER(&pCpuStats->StatR3SyncPTFailed, "/PGM/CPU%u/R3/SyncPT/Failed", "The number of times pfnSyncPT() failed.");
1949	PGM_REG_COUNTER(&pCpuStats->StatR3SyncPT4K, "/PGM/CPU%u/R3/SyncPT/4K", "Nr of 4K PT syncs");
1950	PGM_REG_COUNTER(&pCpuStats->StatR3SyncPT4M, "/PGM/CPU%u/R3/SyncPT/4M", "Nr of 4M PT syncs");
1951	PGM_REG_COUNTER(&pCpuStats->StatR3SyncPagePDNAs, "/PGM/CPU%u/R3/SyncPagePDNAs", "The number of time we've marked a PD not present from SyncPage to virtualize the accessed bit.");
1952	PGM_REG_COUNTER(&pCpuStats->StatR3SyncPagePDOutOfSync, "/PGM/CPU%u/R3/SyncPagePDOutOfSync", "The number of time we've encountered an out-of-sync PD in SyncPage.");
1953	PGM_REG_COUNTER(&pCpuStats->StatR3AccessedPage, "/PGM/CPU%u/R3/AccessedPage", "The number of pages marked not present for accessed bit emulation.");
1954	PGM_REG_PROFILE(&pCpuStats->StatR3DirtyBitTracking, "/PGM/CPU%u/R3/DirtyPage", "Profiling the dirty bit tracking in CheckPageFault().");
1955	PGM_REG_COUNTER(&pCpuStats->StatR3DirtyPage, "/PGM/CPU%u/R3/DirtyPage/Mark", "The number of pages marked read-only for dirty bit tracking.");
1956	PGM_REG_COUNTER(&pCpuStats->StatR3DirtyPageBig, "/PGM/CPU%u/R3/DirtyPage/MarkBig", "The number of 4MB pages marked read-only for dirty bit tracking.");
1957	PGM_REG_COUNTER(&pCpuStats->StatR3DirtyPageSkipped, "/PGM/CPU%u/R3/DirtyPage/Skipped", "The number of pages already dirty or readonly.");
1958	PGM_REG_COUNTER(&pCpuStats->StatR3DirtyPageTrap, "/PGM/CPU%u/R3/DirtyPage/Trap", "The number of traps generated for dirty bit tracking.");
1959	PGM_REG_COUNTER(&pCpuStats->StatR3DirtiedPage, "/PGM/CPU%u/R3/DirtyPage/SetDirty", "The number of pages marked dirty because of write accesses.");
1960	PGM_REG_COUNTER(&pCpuStats->StatR3DirtyTrackRealPF, "/PGM/CPU%u/R3/DirtyPage/RealPF", "The number of real pages faults during dirty bit tracking.");
1961	PGM_REG_COUNTER(&pCpuStats->StatR3PageAlreadyDirty, "/PGM/CPU%u/R3/DirtyPage/AlreadySet", "The number of pages already marked dirty because of write accesses.");
1962	PGM_REG_PROFILE(&pCpuStats->StatR3InvalidatePage, "/PGM/CPU%u/R3/InvalidatePage", "PGMInvalidatePage() profiling.");
1963	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePage4KBPages, "/PGM/CPU%u/R3/InvalidatePage/4KBPages", "The number of times PGMInvalidatePage() was called for a 4KB page.");
1964	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePage4MBPages, "/PGM/CPU%u/R3/InvalidatePage/4MBPages", "The number of times PGMInvalidatePage() was called for a 4MB page.");
1965	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePage4MBPagesSkip, "/PGM/CPU%u/R3/InvalidatePage/4MBPagesSkip","The number of times PGMInvalidatePage() skipped a 4MB page.");
1966	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePagePDMappings, "/PGM/CPU%u/R3/InvalidatePage/PDMappings", "The number of times PGMInvalidatePage() was called for a page directory containing mappings (no conflict).");
1967	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePagePDNAs, "/PGM/CPU%u/R3/InvalidatePage/PDNAs", "The number of times PGMInvalidatePage() was called for a not accessed page directory.");
1968	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePagePDNPs, "/PGM/CPU%u/R3/InvalidatePage/PDNPs", "The number of times PGMInvalidatePage() was called for a not present page directory.");
1969	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePagePDOutOfSync, "/PGM/CPU%u/R3/InvalidatePage/PDOutOfSync", "The number of times PGMInvalidatePage() was called for an out of sync page directory.");
1970	PGM_REG_COUNTER(&pCpuStats->StatR3InvalidatePageSkipped, "/PGM/CPU%u/R3/InvalidatePage/Skipped", "The number of times PGMInvalidatePage() was skipped due to not present shw or pending pending SyncCR3.");
1971	PGM_REG_COUNTER(&pCpuStats->StatR3PageOutOfSyncSupervisor, "/PGM/CPU%u/R3/OutOfSync/SuperVisor", "Number of traps due to pages out of sync and times VerifyAccessSyncPage calls SyncPage.");
1972	PGM_REG_COUNTER(&pCpuStats->StatR3PageOutOfSyncUser, "/PGM/CPU%u/R3/OutOfSync/User", "Number of traps due to pages out of sync and times VerifyAccessSyncPage calls SyncPage.");
1973	PGM_REG_COUNTER(&pCpuStats->StatR3PageOutOfSyncBallloon, "/PGM/CPU%u/R3/OutOfSync/Balloon", "The number of times a ballooned page was accessed (read).");
1974	PGM_REG_PROFILE(&pCpuStats->StatR3Prefetch, "/PGM/CPU%u/R3/Prefetch", "PGMPrefetchPage profiling.");
1975	PGM_REG_PROFILE(&pCpuStats->StatR3FlushTLB, "/PGM/CPU%u/R3/FlushTLB", "Profiling of the PGMFlushTLB() body.");
1976	PGM_REG_COUNTER(&pCpuStats->StatR3FlushTLBNewCR3, "/PGM/CPU%u/R3/FlushTLB/NewCR3", "The number of times PGMFlushTLB was called with a new CR3, non-global. (switch)");
1977	PGM_REG_COUNTER(&pCpuStats->StatR3FlushTLBNewCR3Global, "/PGM/CPU%u/R3/FlushTLB/NewCR3Global", "The number of times PGMFlushTLB was called with a new CR3, global. (switch)");
1978	PGM_REG_COUNTER(&pCpuStats->StatR3FlushTLBSameCR3, "/PGM/CPU%u/R3/FlushTLB/SameCR3", "The number of times PGMFlushTLB was called with the same CR3, non-global. (flush)");
1979	PGM_REG_COUNTER(&pCpuStats->StatR3FlushTLBSameCR3Global, "/PGM/CPU%u/R3/FlushTLB/SameCR3Global", "The number of times PGMFlushTLB was called with the same CR3, global. (flush)");
1980	PGM_REG_PROFILE(&pCpuStats->StatR3GstModifyPage, "/PGM/CPU%u/R3/GstModifyPage", "Profiling of the PGMGstModifyPage() body.");
1981	#endif /* VBOX_WITH_STATISTICS */
1982
1983	#undef PGM_REG_PROFILE
1984	#undef PGM_REG_COUNTER
1985
1986	}
1987
1988	return VINF_SUCCESS;
1989	}
1990
1991
1992	/**
1993	* Init the PGM bits that rely on VMMR0 and MM to be fully initialized.
1994	*
1995	* The dynamic mapping area will also be allocated and initialized at this
1996	* time. We could allocate it during PGMR3Init of course, but the mapping
1997	* wouldn't be allocated at that time preventing us from setting up the
1998	* page table entries with the dummy page.
1999	*
2000	* @returns VBox status code.
2001	* @param pVM VM handle.
2002	*/
2003	VMMR3DECL(int) PGMR3InitDynMap(PVM pVM)
2004	{
2005	RTGCPTR GCPtr;
2006	int rc;
2007
2008	/*
2009	* Reserve space for the dynamic mappings.
2010	*/
2011	rc = MMR3HyperReserve(pVM, MM_HYPER_DYNAMIC_SIZE, "Dynamic mapping", &GCPtr);
2012	if (RT_SUCCESS(rc))
2013	pVM->pgm.s.pbDynPageMapBaseGC = GCPtr;
2014
2015	if ( RT_SUCCESS(rc)
2016	&& (pVM->pgm.s.pbDynPageMapBaseGC >> X86_PD_PAE_SHIFT) != ((pVM->pgm.s.pbDynPageMapBaseGC + MM_HYPER_DYNAMIC_SIZE - 1) >> X86_PD_PAE_SHIFT))
2017	{
2018	rc = MMR3HyperReserve(pVM, MM_HYPER_DYNAMIC_SIZE, "Dynamic mapping not crossing", &GCPtr);
2019	if (RT_SUCCESS(rc))
2020	pVM->pgm.s.pbDynPageMapBaseGC = GCPtr;
2021	}
2022	if (RT_SUCCESS(rc))
2023	{
2024	AssertRelease((pVM->pgm.s.pbDynPageMapBaseGC >> X86_PD_PAE_SHIFT) == ((pVM->pgm.s.pbDynPageMapBaseGC + MM_HYPER_DYNAMIC_SIZE - 1) >> X86_PD_PAE_SHIFT));
2025	MMR3HyperReserve(pVM, PAGE_SIZE, "fence", NULL);
2026	}
2027	return rc;
2028	}
2029
2030
2031	/**
2032	* Ring-3 init finalizing.
2033	*
2034	* @returns VBox status code.
2035	* @param pVM The VM handle.
2036	*/
2037	VMMR3DECL(int) PGMR3InitFinalize(PVM pVM)
2038	{
2039	int rc;
2040
2041	/*
2042	* Reserve space for the dynamic mappings.
2043	* Initialize the dynamic mapping pages with dummy pages to simply the cache.
2044	*/
2045	/* get the pointer to the page table entries. */
2046	PPGMMAPPING pMapping = pgmGetMapping(pVM, pVM->pgm.s.pbDynPageMapBaseGC);
2047	AssertRelease(pMapping);
2048	const uintptr_t off = pVM->pgm.s.pbDynPageMapBaseGC - pMapping->GCPtr;
2049	const unsigned iPT = off >> X86_PD_SHIFT;
2050	const unsigned iPG = (off >> X86_PT_SHIFT) & X86_PT_MASK;
2051	pVM->pgm.s.paDynPageMap32BitPTEsGC = pMapping->aPTs[iPT].pPTRC + iPG * sizeof(pMapping->aPTs[0].pPTR3->a[0]);
2052	pVM->pgm.s.paDynPageMapPaePTEsGC = pMapping->aPTs[iPT].paPaePTsRC + iPG * sizeof(pMapping->aPTs[0].paPaePTsR3->a[0]);
2053
2054	/* init cache area */
2055	RTHCPHYS HCPhysDummy = MMR3PageDummyHCPhys(pVM);
2056	for (uint32_t offDynMap = 0; offDynMap < MM_HYPER_DYNAMIC_SIZE; offDynMap += PAGE_SIZE)
2057	{
2058	rc = PGMMap(pVM, pVM->pgm.s.pbDynPageMapBaseGC + offDynMap, HCPhysDummy, PAGE_SIZE, 0);
2059	AssertRCReturn(rc, rc);
2060	}
2061
2062	/*
2063	* Determine the max physical address width (MAXPHYADDR) and apply it to
2064	* all the mask members and stuff.
2065	*/
2066	uint32_t cMaxPhysAddrWidth;
2067	uint32_t uMaxExtLeaf = ASMCpuId_EAX(0x80000000);
2068	if ( uMaxExtLeaf >= 0x80000008
2069	&& uMaxExtLeaf <= 0x80000fff)
2070	{
2071	cMaxPhysAddrWidth = ASMCpuId_EAX(0x80000008) & 0xff;
2072	LogRel(("PGM: The CPU physical address width is %u bits\n", cMaxPhysAddrWidth));
2073	cMaxPhysAddrWidth = RT_MIN(52, cMaxPhysAddrWidth);
2074	pVM->pgm.s.fLessThan52PhysicalAddressBits = cMaxPhysAddrWidth < 52;
2075	for (uint32_t iBit = cMaxPhysAddrWidth; iBit < 52; iBit++)
2076	pVM->pgm.s.HCPhysInvMmioPg \|= RT_BIT_64(iBit);
2077	}
2078	else
2079	{
2080	LogRel(("PGM: ASSUMING CPU physical address width of 48 bits (uMaxExtLeaf=%#x)\n", uMaxExtLeaf));
2081	cMaxPhysAddrWidth = 48;
2082	pVM->pgm.s.fLessThan52PhysicalAddressBits = true;
2083	pVM->pgm.s.HCPhysInvMmioPg \|= UINT64_C(0x000f0000000000);
2084	}
2085
2086	pVM->pgm.s.GCPhysInvAddrMask = 0;
2087	for (uint32_t iBit = cMaxPhysAddrWidth; iBit < 64; iBit++)
2088	pVM->pgm.s.GCPhysInvAddrMask \|= RT_BIT_64(iBit);
2089
2090	/*
2091	* Initialize the invalid paging entry masks, assuming NX is disabled.
2092	*/
2093	uint64_t fMbzPageFrameMask = pVM->pgm.s.GCPhysInvAddrMask & UINT64_C(0x000ffffffffff000);
2094	for (VMCPUID iCpu = 0; iCpu < pVM->cCpus; iCpu++)
2095	{
2096	PVMCPU pVCpu = &pVM->aCpus[iCpu];
2097
2098	/** @todo The manuals are not entirely clear whether the physical
2099	* address width is relevant. See table 5-9 in the intel
2100	* manual vs the PDE4M descriptions. Write testcase (NP). */
2101	pVCpu->pgm.s.fGst32BitMbzBigPdeMask = ((uint32_t)(fMbzPageFrameMask >> (32 - 13)) & X86_PDE4M_PG_HIGH_MASK)
2102	\| X86_PDE4M_MBZ_MASK;
2103
2104	pVCpu->pgm.s.fGstPaeMbzPteMask = fMbzPageFrameMask \| X86_PTE_PAE_MBZ_MASK_NO_NX;
2105	pVCpu->pgm.s.fGstPaeMbzPdeMask = fMbzPageFrameMask \| X86_PDE_PAE_MBZ_MASK_NO_NX;
2106	pVCpu->pgm.s.fGstPaeMbzBigPdeMask = fMbzPageFrameMask \| X86_PDE2M_PAE_MBZ_MASK_NO_NX;
2107	pVCpu->pgm.s.fGstPaeMbzPdpeMask = fMbzPageFrameMask \| X86_PDPE_PAE_MBZ_MASK;
2108
2109	pVCpu->pgm.s.fGstAmd64MbzPteMask = fMbzPageFrameMask \| X86_PTE_LM_MBZ_MASK_NO_NX;
2110	pVCpu->pgm.s.fGstAmd64MbzPdeMask = fMbzPageFrameMask \| X86_PDE_LM_MBZ_MASK_NX;
2111	pVCpu->pgm.s.fGstAmd64MbzBigPdeMask = fMbzPageFrameMask \| X86_PDE2M_LM_MBZ_MASK_NX;
2112	pVCpu->pgm.s.fGstAmd64MbzPdpeMask = fMbzPageFrameMask \| X86_PDPE_LM_MBZ_MASK_NO_NX;
2113	pVCpu->pgm.s.fGstAmd64MbzBigPdpeMask = fMbzPageFrameMask \| X86_PDPE1G_LM_MBZ_MASK_NO_NX;
2114	pVCpu->pgm.s.fGstAmd64MbzPml4eMask = fMbzPageFrameMask \| X86_PML4E_MBZ_MASK_NO_NX;
2115
2116	pVCpu->pgm.s.fGst64ShadowedPteMask = X86_PTE_P \| X86_PTE_RW \| X86_PTE_US \| X86_PTE_G \| X86_PTE_A \| X86_PTE_D;
2117	pVCpu->pgm.s.fGst64ShadowedPdeMask = X86_PDE_P \| X86_PDE_RW \| X86_PDE_US \| X86_PDE_A;
2118	pVCpu->pgm.s.fGst64ShadowedBigPdeMask = X86_PDE4M_P \| X86_PDE4M_RW \| X86_PDE4M_US \| X86_PDE4M_A;
2119	pVCpu->pgm.s.fGst64ShadowedBigPde4PteMask =
2120	X86_PDE4M_P \| X86_PDE4M_RW \| X86_PDE4M_US \| X86_PDE4M_G \| X86_PDE4M_A \| X86_PDE4M_D;
2121	pVCpu->pgm.s.fGstAmd64ShadowedPdpeMask = X86_PDPE_P \| X86_PDPE_RW \| X86_PDPE_US \| X86_PDPE_A;
2122	pVCpu->pgm.s.fGstAmd64ShadowedPml4eMask = X86_PML4E_P \| X86_PML4E_RW \| X86_PML4E_US \| X86_PML4E_A;
2123	}
2124
2125	/*
2126	* Note that AMD uses all the 8 reserved bits for the address (so 40 bits in total);
2127	* Intel only goes up to 36 bits, so we stick to 36 as well.
2128	* Update: More recent intel manuals specifies 40 bits just like AMD.
2129	*/
2130	uint32_t u32Dummy, u32Features;
2131	CPUMGetGuestCpuId(VMMGetCpu(pVM), 1, &u32Dummy, &u32Dummy, &u32Dummy, &u32Features);
2132	if (u32Features & X86_CPUID_FEATURE_EDX_PSE36)
2133	pVM->pgm.s.GCPhys4MBPSEMask = RT_BIT_64(RT_MAX(36, cMaxPhysAddrWidth)) - 1;
2134	else
2135	pVM->pgm.s.GCPhys4MBPSEMask = RT_BIT_64(32) - 1;
2136
2137	/*
2138	* Allocate memory if we're supposed to do that.
2139	*/
2140	if (pVM->pgm.s.fRamPreAlloc)
2141	rc = pgmR3PhysRamPreAllocate(pVM);
2142
2143	LogRel(("PGMR3InitFinalize: 4 MB PSE mask %RGp\n", pVM->pgm.s.GCPhys4MBPSEMask));
2144	return rc;
2145	}
2146
2147
2148	/**
2149	* Init phase completed callback.
2150	*
2151	* @returns VBox status code.
2152	* @param pVM The VM handle.
2153	* @param enmWhat What has been completed.
2154	* @thread EMT(0)
2155	*/
2156	VMMR3_INT_DECL(int) PGMR3InitCompleted(PVM pVM, VMINITCOMPLETED enmWhat)
2157	{
2158	switch (enmWhat)
2159	{
2160	case VMINITCOMPLETED_HWACCM:
2161	#ifdef VBOX_WITH_PCI_PASSTHROUGH
2162	if (pVM->pgm.s.fPciPassthrough)
2163	{
2164	AssertLogRelReturn(pVM->pgm.s.fRamPreAlloc, VERR_INVALID_PARAMETER);
2165	AssertLogRelReturn(HWACCMIsEnabled(pVM), VERR_INVALID_PARAMETER);
2166	AssertLogRelReturn(HWACCMIsNestedPagingActive(pVM), VERR_INVALID_PARAMETER);
2167
2168	/*
2169	* Report assignments to the IOMMU (hope that's good enough for now).
2170	*/
2171	if (pVM->pgm.s.fPciPassthrough)
2172	{
2173	int rc = VMMR3CallR0(pVM, VMMR0_DO_PGM_PHYS_SETUP_IOMMU, 0, NULL);
2174	AssertRCReturn(rc, rc);
2175	}
2176	}
2177	#else
2178	AssertLogRelReturn(!pVM->pgm.s.fPciPassthrough, VERR_INTERNAL_ERROR_5);
2179	#endif
2180	break;
2181
2182	default:
2183	/* shut up gcc */
2184	break;
2185	}
2186
2187	return VINF_SUCCESS;
2188	}
2189
2190
2191	/**
2192	* Applies relocations to data and code managed by this component.
2193	*
2194	* This function will be called at init and whenever the VMM need to relocate it
2195	* self inside the GC.
2196	*
2197	* @param pVM The VM.
2198	* @param offDelta Relocation delta relative to old location.
2199	*/
2200	VMMR3DECL(void) PGMR3Relocate(PVM pVM, RTGCINTPTR offDelta)
2201	{
2202	LogFlow(("PGMR3Relocate %RGv to %RGv\n", pVM->pgm.s.GCPtrCR3Mapping, pVM->pgm.s.GCPtrCR3Mapping + offDelta));
2203
2204	/*
2205	* Paging stuff.
2206	*/
2207	pVM->pgm.s.GCPtrCR3Mapping += offDelta;
2208
2209	pgmR3ModeDataInit(pVM, true /* resolve GC/R0 symbols */);
2210
2211	/* Shadow, guest and both mode switch & relocation for each VCPU. */
2212	for (VMCPUID i = 0; i < pVM->cCpus; i++)
2213	{
2214	PVMCPU pVCpu = &pVM->aCpus[i];
2215
2216	pgmR3ModeDataSwitch(pVM, pVCpu, pVCpu->pgm.s.enmShadowMode, pVCpu->pgm.s.enmGuestMode);
2217
2218	PGM_SHW_PFN(Relocate, pVCpu)(pVCpu, offDelta);
2219	PGM_GST_PFN(Relocate, pVCpu)(pVCpu, offDelta);
2220	PGM_BTH_PFN(Relocate, pVCpu)(pVCpu, offDelta);
2221	}
2222
2223	/*
2224	* Trees.
2225	*/
2226	pVM->pgm.s.pTreesRC = MMHyperR3ToRC(pVM, pVM->pgm.s.pTreesR3);
2227
2228	/*
2229	* Ram ranges.
2230	*/
2231	if (pVM->pgm.s.pRamRangesXR3)
2232	{
2233	/* Update the pSelfRC pointers and relink them. */
2234	for (PPGMRAMRANGE pCur = pVM->pgm.s.pRamRangesXR3; pCur; pCur = pCur->pNextR3)
2235	if (!(pCur->fFlags & PGM_RAM_RANGE_FLAGS_FLOATING))
2236	pCur->pSelfRC = MMHyperCCToRC(pVM, pCur);
2237	pgmR3PhysRelinkRamRanges(pVM);
2238
2239	/* Flush the RC TLB. */
2240	for (unsigned i = 0; i < PGM_RAMRANGE_TLB_ENTRIES; i++)
2241	pVM->pgm.s.apRamRangesTlbRC[i] = NIL_RTRCPTR;
2242	}
2243
2244	/*
2245	* Update the pSelfRC pointer of the MMIO2 ram ranges since they might not
2246	* be mapped and thus not included in the above exercise.
2247	*/
2248	for (PPGMMMIO2RANGE pCur = pVM->pgm.s.pMmio2RangesR3; pCur; pCur = pCur->pNextR3)
2249	if (!(pCur->RamRange.fFlags & PGM_RAM_RANGE_FLAGS_FLOATING))
2250	pCur->RamRange.pSelfRC = MMHyperCCToRC(pVM, &pCur->RamRange);
2251
2252	/*
2253	* Update the two page directories with all page table mappings.
2254	* (One or more of them have changed, that's why we're here.)
2255	*/
2256	pVM->pgm.s.pMappingsRC = MMHyperR3ToRC(pVM, pVM->pgm.s.pMappingsR3);
2257	for (PPGMMAPPING pCur = pVM->pgm.s.pMappingsR3; pCur->pNextR3; pCur = pCur->pNextR3)
2258	pCur->pNextRC = MMHyperR3ToRC(pVM, pCur->pNextR3);
2259
2260	/* Relocate GC addresses of Page Tables. */
2261	for (PPGMMAPPING pCur = pVM->pgm.s.pMappingsR3; pCur; pCur = pCur->pNextR3)
2262	{
2263	for (RTHCUINT i = 0; i < pCur->cPTs; i++)
2264	{
2265	pCur->aPTs[i].pPTRC = MMHyperR3ToRC(pVM, pCur->aPTs[i].pPTR3);
2266	pCur->aPTs[i].paPaePTsRC = MMHyperR3ToRC(pVM, pCur->aPTs[i].paPaePTsR3);
2267	}
2268	}
2269
2270	/*
2271	* Dynamic page mapping area.
2272	*/
2273	pVM->pgm.s.paDynPageMap32BitPTEsGC += offDelta;
2274	pVM->pgm.s.paDynPageMapPaePTEsGC += offDelta;
2275	pVM->pgm.s.pbDynPageMapBaseGC += offDelta;
2276
2277	if (pVM->pgm.s.pRCDynMap)
2278	{
2279	pVM->pgm.s.pRCDynMap += offDelta;
2280	PPGMRCDYNMAP pDynMap = (PPGMRCDYNMAP)MMHyperRCToCC(pVM, pVM->pgm.s.pRCDynMap);
2281
2282	pDynMap->paPages += offDelta;
2283	PPGMRCDYNMAPENTRY paPages = (PPGMRCDYNMAPENTRY)MMHyperRCToCC(pVM, pDynMap->paPages);
2284
2285	for (uint32_t iPage = 0; iPage < pDynMap->cPages; iPage++)
2286	{
2287	paPages[iPage].pvPage += offDelta;
2288	paPages[iPage].uPte.pLegacy += offDelta;
2289	paPages[iPage].uPte.pPae += offDelta;
2290	}
2291	}
2292
2293	/*
2294	* The Zero page.
2295	*/
2296	pVM->pgm.s.pvZeroPgR0 = MMHyperR3ToR0(pVM, pVM->pgm.s.pvZeroPgR3);
2297	#ifdef VBOX_WITH_2X_4GB_ADDR_SPACE
2298	AssertRelease(pVM->pgm.s.pvZeroPgR0 != NIL_RTR0PTR \|\| !VMMIsHwVirtExtForced(pVM));
2299	#else
2300	AssertRelease(pVM->pgm.s.pvZeroPgR0 != NIL_RTR0PTR);
2301	#endif
2302
2303	/*
2304	* Physical and virtual handlers.
2305	*/
2306	RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesR3->PhysHandlers, true, pgmR3RelocatePhysHandler, &offDelta);
2307	pVM->pgm.s.pLastPhysHandlerRC = NIL_RTRCPTR;
2308	RTAvlroGCPtrDoWithAll(&pVM->pgm.s.pTreesR3->VirtHandlers, true, pgmR3RelocateVirtHandler, &offDelta);
2309	RTAvlroGCPtrDoWithAll(&pVM->pgm.s.pTreesR3->HyperVirtHandlers, true, pgmR3RelocateHyperVirtHandler, &offDelta);
2310
2311	/*
2312	* The page pool.
2313	*/
2314	pgmR3PoolRelocate(pVM);
2315
2316	#ifdef VBOX_WITH_STATISTICS
2317	/*
2318	* Statistics.
2319	*/
2320	pVM->pgm.s.pStatsRC = MMHyperCCToRC(pVM, pVM->pgm.s.pStatsR3);
2321	for (VMCPUID iCpu = 0; iCpu < pVM->cCpus; iCpu++)
2322	pVM->aCpus[iCpu].pgm.s.pStatsRC = MMHyperCCToRC(pVM, pVM->aCpus[iCpu].pgm.s.pStatsR3);
2323	#endif
2324	}
2325
2326
2327	/**
2328	* Callback function for relocating a physical access handler.
2329	*
2330	* @returns 0 (continue enum)
2331	* @param pNode Pointer to a PGMPHYSHANDLER node.
2332	* @param pvUser Pointer to the offDelta. This is a pointer to the delta since we're
2333	* not certain the delta will fit in a void pointer for all possible configs.
2334	*/
2335	static DECLCALLBACK(int) pgmR3RelocatePhysHandler(PAVLROGCPHYSNODECORE pNode, void *pvUser)
2336	{
2337	PPGMPHYSHANDLER pHandler = (PPGMPHYSHANDLER)pNode;
2338	RTGCINTPTR offDelta = *(PRTGCINTPTR)pvUser;
2339	if (pHandler->pfnHandlerRC)
2340	pHandler->pfnHandlerRC += offDelta;
2341	if (pHandler->pvUserRC >= 0x10000)
2342	pHandler->pvUserRC += offDelta;
2343	return 0;
2344	}
2345
2346
2347	/**
2348	* Callback function for relocating a virtual access handler.
2349	*
2350	* @returns 0 (continue enum)
2351	* @param pNode Pointer to a PGMVIRTHANDLER node.
2352	* @param pvUser Pointer to the offDelta. This is a pointer to the delta since we're
2353	* not certain the delta will fit in a void pointer for all possible configs.
2354	*/
2355	static DECLCALLBACK(int) pgmR3RelocateVirtHandler(PAVLROGCPTRNODECORE pNode, void *pvUser)
2356	{
2357	PPGMVIRTHANDLER pHandler = (PPGMVIRTHANDLER)pNode;
2358	RTGCINTPTR offDelta = *(PRTGCINTPTR)pvUser;
2359	Assert( pHandler->enmType == PGMVIRTHANDLERTYPE_ALL
2360	\|\| pHandler->enmType == PGMVIRTHANDLERTYPE_WRITE);
2361	Assert(pHandler->pfnHandlerRC);
2362	pHandler->pfnHandlerRC += offDelta;
2363	return 0;
2364	}
2365
2366
2367	/**
2368	* Callback function for relocating a virtual access handler for the hypervisor mapping.
2369	*
2370	* @returns 0 (continue enum)
2371	* @param pNode Pointer to a PGMVIRTHANDLER node.
2372	* @param pvUser Pointer to the offDelta. This is a pointer to the delta since we're
2373	* not certain the delta will fit in a void pointer for all possible configs.
2374	*/
2375	static DECLCALLBACK(int) pgmR3RelocateHyperVirtHandler(PAVLROGCPTRNODECORE pNode, void *pvUser)
2376	{
2377	PPGMVIRTHANDLER pHandler = (PPGMVIRTHANDLER)pNode;
2378	RTGCINTPTR offDelta = *(PRTGCINTPTR)pvUser;
2379	Assert(pHandler->enmType == PGMVIRTHANDLERTYPE_HYPERVISOR);
2380	Assert(pHandler->pfnHandlerRC);
2381	pHandler->pfnHandlerRC += offDelta;
2382	return 0;
2383	}
2384
2385
2386	/**
2387	* Resets a virtual CPU when unplugged.
2388	*
2389	* @param pVM The VM handle.
2390	* @param pVCpu The virtual CPU handle.
2391	*/
2392	VMMR3DECL(void) PGMR3ResetUnpluggedCpu(PVM pVM, PVMCPU pVCpu)
2393	{
2394	int rc = PGM_GST_PFN(Exit, pVCpu)(pVCpu);
2395	AssertRC(rc);
2396
2397	rc = PGMR3ChangeMode(pVM, pVCpu, PGMMODE_REAL);
2398	AssertRC(rc);
2399
2400	STAM_REL_COUNTER_RESET(&pVCpu->pgm.s.cGuestModeChanges);
2401
2402	pgmR3PoolResetUnpluggedCpu(pVM, pVCpu);
2403
2404	/*
2405	* Re-init other members.
2406	*/
2407	pVCpu->pgm.s.fA20Enabled = true;
2408
2409	/*
2410	* Clear the FFs PGM owns.
2411	*/
2412	VMCPU_FF_CLEAR(pVCpu, VMCPU_FF_PGM_SYNC_CR3);
2413	VMCPU_FF_CLEAR(pVCpu, VMCPU_FF_PGM_SYNC_CR3_NON_GLOBAL);
2414	}
2415
2416
2417	/**
2418	* The VM is being reset.
2419	*
2420	* For the PGM component this means that any PD write monitors
2421	* needs to be removed.
2422	*
2423	* @param pVM VM handle.
2424	*/
2425	VMMR3DECL(void) PGMR3Reset(PVM pVM)
2426	{
2427	int rc;
2428
2429	LogFlow(("PGMR3Reset:\n"));
2430	VM_ASSERT_EMT(pVM);
2431
2432	pgmLock(pVM);
2433
2434	/*
2435	* Unfix any fixed mappings and disable CR3 monitoring.
2436	*/
2437	pVM->pgm.s.fMappingsFixed = false;
2438	pVM->pgm.s.fMappingsFixedRestored = false;
2439	pVM->pgm.s.GCPtrMappingFixed = NIL_RTGCPTR;
2440	pVM->pgm.s.cbMappingFixed = 0;
2441
2442	/*
2443	* Exit the guest paging mode before the pgm pool gets reset.
2444	* Important to clean up the amd64 case.
2445	*/
2446	for (VMCPUID i = 0; i < pVM->cCpus; i++)
2447	{
2448	PVMCPU pVCpu = &pVM->aCpus[i];
2449	rc = PGM_GST_PFN(Exit, pVCpu)(pVCpu);
2450	AssertRC(rc);
2451	}
2452
2453	#ifdef DEBUG
2454	DBGFR3InfoLog(pVM, "mappings", NULL);
2455	DBGFR3InfoLog(pVM, "handlers", "all nostat");
2456	#endif
2457
2458	/*
2459	* Switch mode back to real mode. (before resetting the pgm pool!)
2460	*/
2461	for (VMCPUID i = 0; i < pVM->cCpus; i++)
2462	{
2463	PVMCPU pVCpu = &pVM->aCpus[i];
2464
2465	rc = PGMR3ChangeMode(pVM, pVCpu, PGMMODE_REAL);
2466	AssertRC(rc);
2467
2468	STAM_REL_COUNTER_RESET(&pVCpu->pgm.s.cGuestModeChanges);
2469	}
2470
2471	/*
2472	* Reset the shadow page pool.
2473	*/
2474	pgmR3PoolReset(pVM);
2475
2476	/*
2477	* Re-init various other members and clear the FFs that PGM owns.
2478	*/
2479	for (VMCPUID i = 0; i < pVM->cCpus; i++)
2480	{
2481	PVMCPU pVCpu = &pVM->aCpus[i];
2482
2483	pVCpu->pgm.s.fA20Enabled = true;
2484	pVCpu->pgm.s.fGst32BitPageSizeExtension = false;
2485	PGMNotifyNxeChanged(pVCpu, false);
2486
2487	VMCPU_FF_CLEAR(pVCpu, VMCPU_FF_PGM_SYNC_CR3);
2488	VMCPU_FF_CLEAR(pVCpu, VMCPU_FF_PGM_SYNC_CR3_NON_GLOBAL);
2489	}
2490
2491	/*
2492	* Reset (zero) RAM and shadow ROM pages.
2493	*/
2494	rc = pgmR3PhysRamReset(pVM);
2495	if (RT_SUCCESS(rc))
2496	rc = pgmR3PhysRomReset(pVM);
2497
2498
2499	pgmUnlock(pVM);
2500	AssertReleaseRC(rc);
2501	}
2502
2503
2504	#ifdef VBOX_STRICT
2505	/**
2506	* VM state change callback for clearing fNoMorePhysWrites after
2507	* a snapshot has been created.
2508	*/
2509	static DECLCALLBACK(void) pgmR3ResetNoMorePhysWritesFlag(PVM pVM, VMSTATE enmState, VMSTATE enmOldState, void *pvUser)
2510	{
2511	if ( enmState == VMSTATE_RUNNING
2512	\|\| enmState == VMSTATE_RESUMING)
2513	pVM->pgm.s.fNoMorePhysWrites = false;
2514	}
2515	#endif
2516
2517	/**
2518	* Private API to reset fNoMorePhysWrites.
2519	*/
2520	VMMR3DECL(void) PGMR3ResetNoMorePhysWritesFlag(PVM pVM)
2521	{
2522	pVM->pgm.s.fNoMorePhysWrites = false;
2523	}
2524
2525	/**
2526	* Terminates the PGM.
2527	*
2528	* @returns VBox status code.
2529	* @param pVM Pointer to VM structure.
2530	*/
2531	VMMR3DECL(int) PGMR3Term(PVM pVM)
2532	{
2533	/* Must free shared pages here. */
2534	pgmLock(pVM);
2535	pgmR3PhysRamTerm(pVM);
2536	pgmR3PhysRomTerm(pVM);
2537	pgmUnlock(pVM);
2538
2539	PGMDeregisterStringFormatTypes();
2540	return PDMR3CritSectDelete(&pVM->pgm.s.CritSect);
2541	}
2542
2543
2544	/**
2545	* Show paging mode.
2546	*
2547	* @param pVM VM Handle.
2548	* @param pHlp The info helpers.
2549	* @param pszArgs "all" (default), "guest", "shadow" or "host".
2550	*/
2551	static DECLCALLBACK(void) pgmR3InfoMode(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs)
2552	{
2553	/* digest argument. */
2554	bool fGuest, fShadow, fHost;
2555	if (pszArgs)
2556	pszArgs = RTStrStripL(pszArgs);
2557	if (!pszArgs \|\| !*pszArgs \|\| strstr(pszArgs, "all"))
2558	fShadow = fHost = fGuest = true;
2559	else
2560	{
2561	fShadow = fHost = fGuest = false;
2562	if (strstr(pszArgs, "guest"))
2563	fGuest = true;
2564	if (strstr(pszArgs, "shadow"))
2565	fShadow = true;
2566	if (strstr(pszArgs, "host"))
2567	fHost = true;
2568	}
2569
2570	/** @todo SMP support! */
2571	/* print info. */
2572	if (fGuest)
2573	pHlp->pfnPrintf(pHlp, "Guest paging mode: %s, changed %RU64 times, A20 %s\n",
2574	PGMGetModeName(pVM->aCpus[0].pgm.s.enmGuestMode), pVM->aCpus[0].pgm.s.cGuestModeChanges.c,
2575	pVM->aCpus[0].pgm.s.fA20Enabled ? "enabled" : "disabled");
2576	if (fShadow)
2577	pHlp->pfnPrintf(pHlp, "Shadow paging mode: %s\n", PGMGetModeName(pVM->aCpus[0].pgm.s.enmShadowMode));
2578	if (fHost)
2579	{
2580	const char *psz;
2581	switch (pVM->pgm.s.enmHostMode)
2582	{
2583	case SUPPAGINGMODE_INVALID: psz = "invalid"; break;
2584	case SUPPAGINGMODE_32_BIT: psz = "32-bit"; break;
2585	case SUPPAGINGMODE_32_BIT_GLOBAL: psz = "32-bit+G"; break;
2586	case SUPPAGINGMODE_PAE: psz = "PAE"; break;
2587	case SUPPAGINGMODE_PAE_GLOBAL: psz = "PAE+G"; break;
2588	case SUPPAGINGMODE_PAE_NX: psz = "PAE+NX"; break;
2589	case SUPPAGINGMODE_PAE_GLOBAL_NX: psz = "PAE+G+NX"; break;
2590	case SUPPAGINGMODE_AMD64: psz = "AMD64"; break;
2591	case SUPPAGINGMODE_AMD64_GLOBAL: psz = "AMD64+G"; break;
2592	case SUPPAGINGMODE_AMD64_NX: psz = "AMD64+NX"; break;
2593	case SUPPAGINGMODE_AMD64_GLOBAL_NX: psz = "AMD64+G+NX"; break;
2594	default: psz = "unknown"; break;
2595	}
2596	pHlp->pfnPrintf(pHlp, "Host paging mode: %s\n", psz);
2597	}
2598	}
2599
2600
2601	/**
2602	* Dump registered MMIO ranges to the log.
2603	*
2604	* @param pVM VM Handle.
2605	* @param pHlp The info helpers.
2606	* @param pszArgs Arguments, ignored.
2607	*/
2608	static DECLCALLBACK(void) pgmR3PhysInfo(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs)
2609	{
2610	NOREF(pszArgs);
2611	pHlp->pfnPrintf(pHlp,
2612	"RAM ranges (pVM=%p)\n"
2613	"%.s %.s\n",
2614	pVM,
2615	sizeof(RTGCPHYS) * 4 + 1, "GC Phys Range ",
2616	sizeof(RTHCPTR) * 2, "pvHC ");
2617
2618	for (PPGMRAMRANGE pCur = pVM->pgm.s.pRamRangesXR3; pCur; pCur = pCur->pNextR3)
2619	pHlp->pfnPrintf(pHlp,
2620	"%RGp-%RGp %RHv %s\n",
2621	pCur->GCPhys,
2622	pCur->GCPhysLast,
2623	pCur->pvR3,
2624	pCur->pszDesc);
2625	}
2626
2627	/**
2628	* Dump the page directory to the log.
2629	*
2630	* @param pVM VM Handle.
2631	* @param pHlp The info helpers.
2632	* @param pszArgs Arguments, ignored.
2633	*/
2634	static DECLCALLBACK(void) pgmR3InfoCr3(PVM pVM, PCDBGFINFOHLP pHlp, const char *pszArgs)
2635	{
2636	/** @todo SMP support!! */
2637	PVMCPU pVCpu = &pVM->aCpus[0];
2638
2639	/** @todo fix this! Convert the PGMR3DumpHierarchyHC functions to do guest stuff. */
2640	/* Big pages supported? */
2641	const bool fPSE = !!(CPUMGetGuestCR4(pVCpu) & X86_CR4_PSE);
2642
2643	/* Global pages supported? */
2644	const bool fPGE = !!(CPUMGetGuestCR4(pVCpu) & X86_CR4_PGE);
2645
2646	NOREF(pszArgs);
2647
2648	/*
2649	* Get page directory addresses.
2650	*/
2651	PX86PD pPDSrc = pgmGstGet32bitPDPtr(pVCpu);
2652	Assert(pPDSrc);
2653	Assert(PGMPhysGCPhys2R3PtrAssert(pVM, (RTGCPHYS)(CPUMGetGuestCR3(pVCpu) & X86_CR3_PAGE_MASK), sizeof(*pPDSrc)) == pPDSrc);
2654
2655	/*
2656	* Iterate the page directory.
2657	*/
2658	for (unsigned iPD = 0; iPD < RT_ELEMENTS(pPDSrc->a); iPD++)
2659	{
2660	X86PDE PdeSrc = pPDSrc->a[iPD];
2661	if (PdeSrc.n.u1Present)
2662	{
2663	if (PdeSrc.b.u1Size && fPSE)
2664	pHlp->pfnPrintf(pHlp,
2665	"%04X - %RGp P=%d U=%d RW=%d G=%d - BIG\n",
2666	iPD,
2667	pgmGstGet4MBPhysPage(pVM, PdeSrc),
2668	PdeSrc.b.u1Present, PdeSrc.b.u1User, PdeSrc.b.u1Write, PdeSrc.b.u1Global && fPGE);
2669	else
2670	pHlp->pfnPrintf(pHlp,
2671	"%04X - %RGp P=%d U=%d RW=%d [G=%d]\n",
2672	iPD,
2673	(RTGCPHYS)(PdeSrc.u & X86_PDE_PG_MASK),
2674	PdeSrc.n.u1Present, PdeSrc.n.u1User, PdeSrc.n.u1Write, PdeSrc.b.u1Global && fPGE);
2675	}
2676	}
2677	}
2678
2679
2680	/**
2681	* Service a VMMCALLRING3_PGM_LOCK call.
2682	*
2683	* @returns VBox status code.
2684	* @param pVM The VM handle.
2685	*/
2686	VMMR3DECL(int) PGMR3LockCall(PVM pVM)
2687	{
2688	int rc = PDMR3CritSectEnterEx(&pVM->pgm.s.CritSect, true /* fHostCall */);
2689	AssertRC(rc);
2690	return rc;
2691	}
2692
2693
2694	/**
2695	* Converts a PGMMODE value to a PGM_TYPE_* \#define.
2696	*
2697	* @returns PGM_TYPE_*.
2698	* @param pgmMode The mode value to convert.
2699	*/
2700	DECLINLINE(unsigned) pgmModeToType(PGMMODE pgmMode)
2701	{
2702	switch (pgmMode)
2703	{
2704	case PGMMODE_REAL: return PGM_TYPE_REAL;
2705	case PGMMODE_PROTECTED: return PGM_TYPE_PROT;
2706	case PGMMODE_32_BIT: return PGM_TYPE_32BIT;
2707	case PGMMODE_PAE:
2708	case PGMMODE_PAE_NX: return PGM_TYPE_PAE;
2709	case PGMMODE_AMD64:
2710	case PGMMODE_AMD64_NX: return PGM_TYPE_AMD64;
2711	case PGMMODE_NESTED: return PGM_TYPE_NESTED;
2712	case PGMMODE_EPT: return PGM_TYPE_EPT;
2713	default:
2714	AssertFatalMsgFailed(("pgmMode=%d\n", pgmMode));
2715	}
2716	}
2717
2718
2719	/**
2720	* Gets the index into the paging mode data array of a SHW+GST mode.
2721	*
2722	* @returns PGM::paPagingData index.
2723	* @param uShwType The shadow paging mode type.
2724	* @param uGstType The guest paging mode type.
2725	*/
2726	DECLINLINE(unsigned) pgmModeDataIndex(unsigned uShwType, unsigned uGstType)
2727	{
2728	Assert(uShwType >= PGM_TYPE_32BIT && uShwType <= PGM_TYPE_MAX);
2729	Assert(uGstType >= PGM_TYPE_REAL && uGstType <= PGM_TYPE_AMD64);
2730	return (uShwType - PGM_TYPE_32BIT) * (PGM_TYPE_AMD64 - PGM_TYPE_REAL + 1)
2731	+ (uGstType - PGM_TYPE_REAL);
2732	}
2733
2734
2735	/**
2736	* Gets the index into the paging mode data array of a SHW+GST mode.
2737	*
2738	* @returns PGM::paPagingData index.
2739	* @param enmShw The shadow paging mode.
2740	* @param enmGst The guest paging mode.
2741	*/
2742	DECLINLINE(unsigned) pgmModeDataIndexByMode(PGMMODE enmShw, PGMMODE enmGst)
2743	{
2744	Assert(enmShw >= PGMMODE_32_BIT && enmShw <= PGMMODE_MAX);
2745	Assert(enmGst > PGMMODE_INVALID && enmGst < PGMMODE_MAX);
2746	return pgmModeDataIndex(pgmModeToType(enmShw), pgmModeToType(enmGst));
2747	}
2748
2749
2750	/**
2751	* Calculates the max data index.
2752	* @returns The number of entries in the paging data array.
2753	*/
2754	DECLINLINE(unsigned) pgmModeDataMaxIndex(void)
2755	{
2756	return pgmModeDataIndex(PGM_TYPE_MAX, PGM_TYPE_AMD64) + 1;
2757	}
2758
2759
2760	/**
2761	* Initializes the paging mode data kept in PGM::paModeData.
2762	*
2763	* @param pVM The VM handle.
2764	* @param fResolveGCAndR0 Indicate whether or not GC and Ring-0 symbols can be resolved now.
2765	* This is used early in the init process to avoid trouble with PDM
2766	* not being initialized yet.
2767	*/
2768	static int pgmR3ModeDataInit(PVM pVM, bool fResolveGCAndR0)
2769	{
2770	PPGMMODEDATA pModeData;
2771	int rc;
2772
2773	/*
2774	* Allocate the array on the first call.
2775	*/
2776	if (!pVM->pgm.s.paModeData)
2777	{
2778	pVM->pgm.s.paModeData = (PPGMMODEDATA)MMR3HeapAllocZ(pVM, MM_TAG_PGM, sizeof(PGMMODEDATA) * pgmModeDataMaxIndex());
2779	AssertReturn(pVM->pgm.s.paModeData, VERR_NO_MEMORY);
2780	}
2781
2782	/*
2783	* Initialize the array entries.
2784	*/
2785	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_32BIT, PGM_TYPE_REAL)];
2786	pModeData->uShwType = PGM_TYPE_32BIT;
2787	pModeData->uGstType = PGM_TYPE_REAL;
2788	rc = PGM_SHW_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2789	rc = PGM_GST_NAME_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2790	rc = PGM_BTH_NAME_32BIT_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2791
2792	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_32BIT, PGMMODE_PROTECTED)];
2793	pModeData->uShwType = PGM_TYPE_32BIT;
2794	pModeData->uGstType = PGM_TYPE_PROT;
2795	rc = PGM_SHW_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2796	rc = PGM_GST_NAME_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2797	rc = PGM_BTH_NAME_32BIT_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2798
2799	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_32BIT, PGM_TYPE_32BIT)];
2800	pModeData->uShwType = PGM_TYPE_32BIT;
2801	pModeData->uGstType = PGM_TYPE_32BIT;
2802	rc = PGM_SHW_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2803	rc = PGM_GST_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2804	rc = PGM_BTH_NAME_32BIT_32BIT(InitData)(pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2805
2806	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_PAE, PGM_TYPE_REAL)];
2807	pModeData->uShwType = PGM_TYPE_PAE;
2808	pModeData->uGstType = PGM_TYPE_REAL;
2809	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2810	rc = PGM_GST_NAME_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2811	rc = PGM_BTH_NAME_PAE_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2812
2813	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_PAE, PGM_TYPE_PROT)];
2814	pModeData->uShwType = PGM_TYPE_PAE;
2815	pModeData->uGstType = PGM_TYPE_PROT;
2816	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2817	rc = PGM_GST_NAME_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2818	rc = PGM_BTH_NAME_PAE_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2819
2820	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_PAE, PGM_TYPE_32BIT)];
2821	pModeData->uShwType = PGM_TYPE_PAE;
2822	pModeData->uGstType = PGM_TYPE_32BIT;
2823	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2824	rc = PGM_GST_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2825	rc = PGM_BTH_NAME_PAE_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2826
2827	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_PAE, PGM_TYPE_PAE)];
2828	pModeData->uShwType = PGM_TYPE_PAE;
2829	pModeData->uGstType = PGM_TYPE_PAE;
2830	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2831	rc = PGM_GST_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2832	rc = PGM_BTH_NAME_PAE_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2833
2834	#ifdef VBOX_WITH_64_BITS_GUESTS
2835	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_AMD64, PGM_TYPE_AMD64)];
2836	pModeData->uShwType = PGM_TYPE_AMD64;
2837	pModeData->uGstType = PGM_TYPE_AMD64;
2838	rc = PGM_SHW_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2839	rc = PGM_GST_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2840	rc = PGM_BTH_NAME_AMD64_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2841	#endif
2842
2843	/* The nested paging mode. */
2844	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_REAL)];
2845	pModeData->uShwType = PGM_TYPE_NESTED;
2846	pModeData->uGstType = PGM_TYPE_REAL;
2847	rc = PGM_GST_NAME_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2848	rc = PGM_BTH_NAME_NESTED_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2849
2850	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGMMODE_PROTECTED)];
2851	pModeData->uShwType = PGM_TYPE_NESTED;
2852	pModeData->uGstType = PGM_TYPE_PROT;
2853	rc = PGM_GST_NAME_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2854	rc = PGM_BTH_NAME_NESTED_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2855
2856	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_32BIT)];
2857	pModeData->uShwType = PGM_TYPE_NESTED;
2858	pModeData->uGstType = PGM_TYPE_32BIT;
2859	rc = PGM_GST_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2860	rc = PGM_BTH_NAME_NESTED_32BIT(InitData)(pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2861
2862	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_PAE)];
2863	pModeData->uShwType = PGM_TYPE_NESTED;
2864	pModeData->uGstType = PGM_TYPE_PAE;
2865	rc = PGM_GST_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2866	rc = PGM_BTH_NAME_NESTED_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2867
2868	#ifdef VBOX_WITH_64_BITS_GUESTS
2869	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_AMD64)];
2870	pModeData->uShwType = PGM_TYPE_NESTED;
2871	pModeData->uGstType = PGM_TYPE_AMD64;
2872	rc = PGM_GST_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2873	rc = PGM_BTH_NAME_NESTED_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2874	#endif
2875
2876	/* The shadow part of the nested callback mode depends on the host paging mode (AMD-V only). */
2877	switch (pVM->pgm.s.enmHostMode)
2878	{
2879	#if HC_ARCH_BITS == 32
2880	case SUPPAGINGMODE_32_BIT:
2881	case SUPPAGINGMODE_32_BIT_GLOBAL:
2882	for (unsigned i = PGM_TYPE_REAL; i <= PGM_TYPE_PAE; i++)
2883	{
2884	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, i)];
2885	rc = PGM_SHW_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2886	}
2887	# ifdef VBOX_WITH_64_BITS_GUESTS
2888	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_AMD64)];
2889	rc = PGM_SHW_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2890	# endif
2891	break;
2892
2893	case SUPPAGINGMODE_PAE:
2894	case SUPPAGINGMODE_PAE_NX:
2895	case SUPPAGINGMODE_PAE_GLOBAL:
2896	case SUPPAGINGMODE_PAE_GLOBAL_NX:
2897	for (unsigned i = PGM_TYPE_REAL; i <= PGM_TYPE_PAE; i++)
2898	{
2899	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, i)];
2900	rc = PGM_SHW_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2901	}
2902	# ifdef VBOX_WITH_64_BITS_GUESTS
2903	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, PGM_TYPE_AMD64)];
2904	rc = PGM_SHW_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2905	# endif
2906	break;
2907	#endif /* HC_ARCH_BITS == 32 */
2908
2909	#if HC_ARCH_BITS == 64 \|\| defined(RT_OS_DARWIN)
2910	case SUPPAGINGMODE_AMD64:
2911	case SUPPAGINGMODE_AMD64_GLOBAL:
2912	case SUPPAGINGMODE_AMD64_NX:
2913	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
2914	# ifdef VBOX_WITH_64_BITS_GUESTS
2915	for (unsigned i = PGM_TYPE_REAL; i <= PGM_TYPE_AMD64; i++)
2916	# else
2917	for (unsigned i = PGM_TYPE_REAL; i <= PGM_TYPE_PAE; i++)
2918	# endif
2919	{
2920	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_NESTED, i)];
2921	rc = PGM_SHW_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2922	}
2923	break;
2924	#endif /* HC_ARCH_BITS == 64 \|\| RT_OS_DARWIN */
2925
2926	default:
2927	AssertFailed();
2928	break;
2929	}
2930
2931	/* Extended paging (EPT) / Intel VT-x */
2932	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_EPT, PGM_TYPE_REAL)];
2933	pModeData->uShwType = PGM_TYPE_EPT;
2934	pModeData->uGstType = PGM_TYPE_REAL;
2935	rc = PGM_SHW_NAME_EPT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2936	rc = PGM_GST_NAME_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2937	rc = PGM_BTH_NAME_EPT_REAL(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2938
2939	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_EPT, PGM_TYPE_PROT)];
2940	pModeData->uShwType = PGM_TYPE_EPT;
2941	pModeData->uGstType = PGM_TYPE_PROT;
2942	rc = PGM_SHW_NAME_EPT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2943	rc = PGM_GST_NAME_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2944	rc = PGM_BTH_NAME_EPT_PROT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2945
2946	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_EPT, PGM_TYPE_32BIT)];
2947	pModeData->uShwType = PGM_TYPE_EPT;
2948	pModeData->uGstType = PGM_TYPE_32BIT;
2949	rc = PGM_SHW_NAME_EPT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2950	rc = PGM_GST_NAME_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2951	rc = PGM_BTH_NAME_EPT_32BIT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2952
2953	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_EPT, PGM_TYPE_PAE)];
2954	pModeData->uShwType = PGM_TYPE_EPT;
2955	pModeData->uGstType = PGM_TYPE_PAE;
2956	rc = PGM_SHW_NAME_EPT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2957	rc = PGM_GST_NAME_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2958	rc = PGM_BTH_NAME_EPT_PAE(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2959
2960	#ifdef VBOX_WITH_64_BITS_GUESTS
2961	pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndex(PGM_TYPE_EPT, PGM_TYPE_AMD64)];
2962	pModeData->uShwType = PGM_TYPE_EPT;
2963	pModeData->uGstType = PGM_TYPE_AMD64;
2964	rc = PGM_SHW_NAME_EPT(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2965	rc = PGM_GST_NAME_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2966	rc = PGM_BTH_NAME_EPT_AMD64(InitData)( pVM, pModeData, fResolveGCAndR0); AssertRCReturn(rc, rc);
2967	#endif
2968	return VINF_SUCCESS;
2969	}
2970
2971
2972	/**
2973	* Switch to different (or relocated in the relocate case) mode data.
2974	*
2975	* @param pVM The VM handle.
2976	* @param pVCpu The VMCPU to operate on.
2977	* @param enmShw The the shadow paging mode.
2978	* @param enmGst The the guest paging mode.
2979	*/
2980	static void pgmR3ModeDataSwitch(PVM pVM, PVMCPU pVCpu, PGMMODE enmShw, PGMMODE enmGst)
2981	{
2982	PPGMMODEDATA pModeData = &pVM->pgm.s.paModeData[pgmModeDataIndexByMode(enmShw, enmGst)];
2983
2984	Assert(pModeData->uGstType == pgmModeToType(enmGst));
2985	Assert(pModeData->uShwType == pgmModeToType(enmShw));
2986
2987	/* shadow */
2988	pVCpu->pgm.s.pfnR3ShwRelocate = pModeData->pfnR3ShwRelocate;
2989	pVCpu->pgm.s.pfnR3ShwExit = pModeData->pfnR3ShwExit;
2990	pVCpu->pgm.s.pfnR3ShwGetPage = pModeData->pfnR3ShwGetPage;
2991	Assert(pVCpu->pgm.s.pfnR3ShwGetPage);
2992	pVCpu->pgm.s.pfnR3ShwModifyPage = pModeData->pfnR3ShwModifyPage;
2993
2994	pVCpu->pgm.s.pfnRCShwGetPage = pModeData->pfnRCShwGetPage;
2995	pVCpu->pgm.s.pfnRCShwModifyPage = pModeData->pfnRCShwModifyPage;
2996
2997	pVCpu->pgm.s.pfnR0ShwGetPage = pModeData->pfnR0ShwGetPage;
2998	pVCpu->pgm.s.pfnR0ShwModifyPage = pModeData->pfnR0ShwModifyPage;
2999
3000
3001	/* guest */
3002	pVCpu->pgm.s.pfnR3GstRelocate = pModeData->pfnR3GstRelocate;
3003	pVCpu->pgm.s.pfnR3GstExit = pModeData->pfnR3GstExit;
3004	pVCpu->pgm.s.pfnR3GstGetPage = pModeData->pfnR3GstGetPage;
3005	Assert(pVCpu->pgm.s.pfnR3GstGetPage);
3006	pVCpu->pgm.s.pfnR3GstModifyPage = pModeData->pfnR3GstModifyPage;
3007	pVCpu->pgm.s.pfnR3GstGetPDE = pModeData->pfnR3GstGetPDE;
3008	pVCpu->pgm.s.pfnRCGstGetPage = pModeData->pfnRCGstGetPage;
3009	pVCpu->pgm.s.pfnRCGstModifyPage = pModeData->pfnRCGstModifyPage;
3010	pVCpu->pgm.s.pfnRCGstGetPDE = pModeData->pfnRCGstGetPDE;
3011	pVCpu->pgm.s.pfnR0GstGetPage = pModeData->pfnR0GstGetPage;
3012	pVCpu->pgm.s.pfnR0GstModifyPage = pModeData->pfnR0GstModifyPage;
3013	pVCpu->pgm.s.pfnR0GstGetPDE = pModeData->pfnR0GstGetPDE;
3014
3015	/* both */
3016	pVCpu->pgm.s.pfnR3BthRelocate = pModeData->pfnR3BthRelocate;
3017	pVCpu->pgm.s.pfnR3BthInvalidatePage = pModeData->pfnR3BthInvalidatePage;
3018	pVCpu->pgm.s.pfnR3BthSyncCR3 = pModeData->pfnR3BthSyncCR3;
3019	Assert(pVCpu->pgm.s.pfnR3BthSyncCR3);
3020	pVCpu->pgm.s.pfnR3BthPrefetchPage = pModeData->pfnR3BthPrefetchPage;
3021	pVCpu->pgm.s.pfnR3BthVerifyAccessSyncPage = pModeData->pfnR3BthVerifyAccessSyncPage;
3022	#ifdef VBOX_STRICT
3023	pVCpu->pgm.s.pfnR3BthAssertCR3 = pModeData->pfnR3BthAssertCR3;
3024	#endif
3025	pVCpu->pgm.s.pfnR3BthMapCR3 = pModeData->pfnR3BthMapCR3;
3026	pVCpu->pgm.s.pfnR3BthUnmapCR3 = pModeData->pfnR3BthUnmapCR3;
3027
3028	pVCpu->pgm.s.pfnRCBthTrap0eHandler = pModeData->pfnRCBthTrap0eHandler;
3029	pVCpu->pgm.s.pfnRCBthInvalidatePage = pModeData->pfnRCBthInvalidatePage;
3030	pVCpu->pgm.s.pfnRCBthSyncCR3 = pModeData->pfnRCBthSyncCR3;
3031	pVCpu->pgm.s.pfnRCBthPrefetchPage = pModeData->pfnRCBthPrefetchPage;
3032	pVCpu->pgm.s.pfnRCBthVerifyAccessSyncPage = pModeData->pfnRCBthVerifyAccessSyncPage;
3033	#ifdef VBOX_STRICT
3034	pVCpu->pgm.s.pfnRCBthAssertCR3 = pModeData->pfnRCBthAssertCR3;
3035	#endif
3036	pVCpu->pgm.s.pfnRCBthMapCR3 = pModeData->pfnRCBthMapCR3;
3037	pVCpu->pgm.s.pfnRCBthUnmapCR3 = pModeData->pfnRCBthUnmapCR3;
3038
3039	pVCpu->pgm.s.pfnR0BthTrap0eHandler = pModeData->pfnR0BthTrap0eHandler;
3040	pVCpu->pgm.s.pfnR0BthInvalidatePage = pModeData->pfnR0BthInvalidatePage;
3041	pVCpu->pgm.s.pfnR0BthSyncCR3 = pModeData->pfnR0BthSyncCR3;
3042	pVCpu->pgm.s.pfnR0BthPrefetchPage = pModeData->pfnR0BthPrefetchPage;
3043	pVCpu->pgm.s.pfnR0BthVerifyAccessSyncPage = pModeData->pfnR0BthVerifyAccessSyncPage;
3044	#ifdef VBOX_STRICT
3045	pVCpu->pgm.s.pfnR0BthAssertCR3 = pModeData->pfnR0BthAssertCR3;
3046	#endif
3047	pVCpu->pgm.s.pfnR0BthMapCR3 = pModeData->pfnR0BthMapCR3;
3048	pVCpu->pgm.s.pfnR0BthUnmapCR3 = pModeData->pfnR0BthUnmapCR3;
3049	}
3050
3051
3052	/**
3053	* Calculates the shadow paging mode.
3054	*
3055	* @returns The shadow paging mode.
3056	* @param pVM VM handle.
3057	* @param enmGuestMode The guest mode.
3058	* @param enmHostMode The host mode.
3059	* @param enmShadowMode The current shadow mode.
3060	* @param penmSwitcher Where to store the switcher to use.
3061	* VMMSWITCHER_INVALID means no change.
3062	*/
3063	static PGMMODE pgmR3CalcShadowMode(PVM pVM, PGMMODE enmGuestMode, SUPPAGINGMODE enmHostMode, PGMMODE enmShadowMode, VMMSWITCHER *penmSwitcher)
3064	{
3065	VMMSWITCHER enmSwitcher = VMMSWITCHER_INVALID;
3066	switch (enmGuestMode)
3067	{
3068	/*
3069	* When switching to real or protected mode we don't change
3070	* anything since it's likely that we'll switch back pretty soon.
3071	*
3072	* During pgmR3InitPaging we'll end up here with PGMMODE_INVALID
3073	* and is supposed to determine which shadow paging and switcher to
3074	* use during init.
3075	*/
3076	case PGMMODE_REAL:
3077	case PGMMODE_PROTECTED:
3078	if ( enmShadowMode != PGMMODE_INVALID
3079	&& !HWACCMIsEnabled(pVM) /* always switch in hwaccm mode! */)
3080	break; /* (no change) */
3081
3082	switch (enmHostMode)
3083	{
3084	case SUPPAGINGMODE_32_BIT:
3085	case SUPPAGINGMODE_32_BIT_GLOBAL:
3086	enmShadowMode = PGMMODE_32_BIT;
3087	enmSwitcher = VMMSWITCHER_32_TO_32;
3088	break;
3089
3090	case SUPPAGINGMODE_PAE:
3091	case SUPPAGINGMODE_PAE_NX:
3092	case SUPPAGINGMODE_PAE_GLOBAL:
3093	case SUPPAGINGMODE_PAE_GLOBAL_NX:
3094	enmShadowMode = PGMMODE_PAE;
3095	enmSwitcher = VMMSWITCHER_PAE_TO_PAE;
3096	#ifdef DEBUG_bird
3097	if (RTEnvExist("VBOX_32BIT"))
3098	{
3099	enmShadowMode = PGMMODE_32_BIT;
3100	enmSwitcher = VMMSWITCHER_PAE_TO_32;
3101	}
3102	#endif
3103	break;
3104
3105	case SUPPAGINGMODE_AMD64:
3106	case SUPPAGINGMODE_AMD64_GLOBAL:
3107	case SUPPAGINGMODE_AMD64_NX:
3108	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
3109	enmShadowMode = PGMMODE_PAE;
3110	enmSwitcher = VMMSWITCHER_AMD64_TO_PAE;
3111	#ifdef DEBUG_bird
3112	if (RTEnvExist("VBOX_32BIT"))
3113	{
3114	enmShadowMode = PGMMODE_32_BIT;
3115	enmSwitcher = VMMSWITCHER_AMD64_TO_32;
3116	}
3117	#endif
3118	break;
3119
3120	default: AssertMsgFailed(("enmHostMode=%d\n", enmHostMode)); break;
3121	}
3122	break;
3123
3124	case PGMMODE_32_BIT:
3125	switch (enmHostMode)
3126	{
3127	case SUPPAGINGMODE_32_BIT:
3128	case SUPPAGINGMODE_32_BIT_GLOBAL:
3129	enmShadowMode = PGMMODE_32_BIT;
3130	enmSwitcher = VMMSWITCHER_32_TO_32;
3131	break;
3132
3133	case SUPPAGINGMODE_PAE:
3134	case SUPPAGINGMODE_PAE_NX:
3135	case SUPPAGINGMODE_PAE_GLOBAL:
3136	case SUPPAGINGMODE_PAE_GLOBAL_NX:
3137	enmShadowMode = PGMMODE_PAE;
3138	enmSwitcher = VMMSWITCHER_PAE_TO_PAE;
3139	#ifdef DEBUG_bird
3140	if (RTEnvExist("VBOX_32BIT"))
3141	{
3142	enmShadowMode = PGMMODE_32_BIT;
3143	enmSwitcher = VMMSWITCHER_PAE_TO_32;
3144	}
3145	#endif
3146	break;
3147
3148	case SUPPAGINGMODE_AMD64:
3149	case SUPPAGINGMODE_AMD64_GLOBAL:
3150	case SUPPAGINGMODE_AMD64_NX:
3151	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
3152	enmShadowMode = PGMMODE_PAE;
3153	enmSwitcher = VMMSWITCHER_AMD64_TO_PAE;
3154	#ifdef DEBUG_bird
3155	if (RTEnvExist("VBOX_32BIT"))
3156	{
3157	enmShadowMode = PGMMODE_32_BIT;
3158	enmSwitcher = VMMSWITCHER_AMD64_TO_32;
3159	}
3160	#endif
3161	break;
3162
3163	default: AssertMsgFailed(("enmHostMode=%d\n", enmHostMode)); break;
3164	}
3165	break;
3166
3167	case PGMMODE_PAE:
3168	case PGMMODE_PAE_NX: /** @todo This might require more switchers and guest+both modes. */
3169	switch (enmHostMode)
3170	{
3171	case SUPPAGINGMODE_32_BIT:
3172	case SUPPAGINGMODE_32_BIT_GLOBAL:
3173	enmShadowMode = PGMMODE_PAE;
3174	enmSwitcher = VMMSWITCHER_32_TO_PAE;
3175	break;
3176
3177	case SUPPAGINGMODE_PAE:
3178	case SUPPAGINGMODE_PAE_NX:
3179	case SUPPAGINGMODE_PAE_GLOBAL:
3180	case SUPPAGINGMODE_PAE_GLOBAL_NX:
3181	enmShadowMode = PGMMODE_PAE;
3182	enmSwitcher = VMMSWITCHER_PAE_TO_PAE;
3183	break;
3184
3185	case SUPPAGINGMODE_AMD64:
3186	case SUPPAGINGMODE_AMD64_GLOBAL:
3187	case SUPPAGINGMODE_AMD64_NX:
3188	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
3189	enmShadowMode = PGMMODE_PAE;
3190	enmSwitcher = VMMSWITCHER_AMD64_TO_PAE;
3191	break;
3192
3193	default: AssertMsgFailed(("enmHostMode=%d\n", enmHostMode)); break;
3194	}
3195	break;
3196
3197	case PGMMODE_AMD64:
3198	case PGMMODE_AMD64_NX:
3199	switch (enmHostMode)
3200	{
3201	case SUPPAGINGMODE_32_BIT:
3202	case SUPPAGINGMODE_32_BIT_GLOBAL:
3203	enmShadowMode = PGMMODE_AMD64;
3204	enmSwitcher = VMMSWITCHER_32_TO_AMD64;
3205	break;
3206
3207	case SUPPAGINGMODE_PAE:
3208	case SUPPAGINGMODE_PAE_NX:
3209	case SUPPAGINGMODE_PAE_GLOBAL:
3210	case SUPPAGINGMODE_PAE_GLOBAL_NX:
3211	enmShadowMode = PGMMODE_AMD64;
3212	enmSwitcher = VMMSWITCHER_PAE_TO_AMD64;
3213	break;
3214
3215	case SUPPAGINGMODE_AMD64:
3216	case SUPPAGINGMODE_AMD64_GLOBAL:
3217	case SUPPAGINGMODE_AMD64_NX:
3218	case SUPPAGINGMODE_AMD64_GLOBAL_NX:
3219	enmShadowMode = PGMMODE_AMD64;
3220	enmSwitcher = VMMSWITCHER_AMD64_TO_AMD64;
3221	break;
3222
3223	default: AssertMsgFailed(("enmHostMode=%d\n", enmHostMode)); break;
3224	}
3225	break;
3226
3227
3228	default:
3229	AssertReleaseMsgFailed(("enmGuestMode=%d\n", enmGuestMode));
3230	*penmSwitcher = VMMSWITCHER_INVALID;
3231	return PGMMODE_INVALID;
3232	}
3233	/* Override the shadow mode is nested paging is active. */
3234	pVM->pgm.s.fNestedPaging = HWACCMIsNestedPagingActive(pVM);
3235	if (pVM->pgm.s.fNestedPaging)
3236	enmShadowMode = HWACCMGetShwPagingMode(pVM);
3237
3238	*penmSwitcher = enmSwitcher;
3239	return enmShadowMode;
3240	}
3241
3242
3243	/**
3244	* Performs the actual mode change.
3245	* This is called by PGMChangeMode and pgmR3InitPaging().
3246	*
3247	* @returns VBox status code. May suspend or power off the VM on error, but this
3248	* will trigger using FFs and not status codes.
3249	*
3250	* @param pVM VM handle.
3251	* @param pVCpu The VMCPU to operate on.
3252	* @param enmGuestMode The new guest mode. This is assumed to be different from
3253	* the current mode.
3254	*/
3255	VMMR3DECL(int) PGMR3ChangeMode(PVM pVM, PVMCPU pVCpu, PGMMODE enmGuestMode)
3256	{
3257	bool fIsOldGuestPagingMode64Bits = (pVCpu->pgm.s.enmGuestMode >= PGMMODE_AMD64);
3258	bool fIsNewGuestPagingMode64Bits = (enmGuestMode >= PGMMODE_AMD64);
3259
3260	Log(("PGMR3ChangeMode: Guest mode: %s -> %s\n", PGMGetModeName(pVCpu->pgm.s.enmGuestMode), PGMGetModeName(enmGuestMode)));
3261	STAM_REL_COUNTER_INC(&pVCpu->pgm.s.cGuestModeChanges);
3262
3263	/*
3264	* Calc the shadow mode and switcher.
3265	*/
3266	VMMSWITCHER enmSwitcher;
3267	PGMMODE enmShadowMode;
3268	enmShadowMode = pgmR3CalcShadowMode(pVM, enmGuestMode, pVM->pgm.s.enmHostMode, pVCpu->pgm.s.enmShadowMode, &enmSwitcher);
3269
3270	#ifdef VBOX_WITH_RAW_MODE
3271	if (enmSwitcher != VMMSWITCHER_INVALID)
3272	{
3273	/*
3274	* Select new switcher.
3275	*/
3276	int rc = VMMR3SelectSwitcher(pVM, enmSwitcher);
3277	if (RT_FAILURE(rc))
3278	{
3279	AssertReleaseMsgFailed(("VMMR3SelectSwitcher(%d) -> %Rrc\n", enmSwitcher, rc));
3280	return rc;
3281	}
3282	}
3283	#endif
3284
3285	/*
3286	* Exit old mode(s).
3287	*/
3288	#if HC_ARCH_BITS == 32
3289	/* The nested shadow paging mode for AMD-V does change when running 64 bits guests on 32 bits hosts; typically PAE <-> AMD64 */
3290	const bool fForceShwEnterExit = ( fIsOldGuestPagingMode64Bits != fIsNewGuestPagingMode64Bits
3291	&& enmShadowMode == PGMMODE_NESTED);
3292	#else
3293	const bool fForceShwEnterExit = false;
3294	#endif
3295	/* shadow */
3296	if ( enmShadowMode != pVCpu->pgm.s.enmShadowMode
3297	\|\| fForceShwEnterExit)
3298	{
3299	LogFlow(("PGMR3ChangeMode: Shadow mode: %s -> %s\n", PGMGetModeName(pVCpu->pgm.s.enmShadowMode), PGMGetModeName(enmShadowMode)));
3300	if (PGM_SHW_PFN(Exit, pVCpu))
3301	{
3302	int rc = PGM_SHW_PFN(Exit, pVCpu)(pVCpu);
3303	if (RT_FAILURE(rc))
3304	{
3305	AssertMsgFailed(("Exit failed for shadow mode %d: %Rrc\n", pVCpu->pgm.s.enmShadowMode, rc));
3306	return rc;
3307	}
3308	}
3309
3310	}
3311	else
3312	LogFlow(("PGMR3ChangeMode: Shadow mode remains: %s\n", PGMGetModeName(pVCpu->pgm.s.enmShadowMode)));
3313
3314	/* guest */
3315	if (PGM_GST_PFN(Exit, pVCpu))
3316	{
3317	int rc = PGM_GST_PFN(Exit, pVCpu)(pVCpu);
3318	if (RT_FAILURE(rc))
3319	{
3320	AssertMsgFailed(("Exit failed for guest mode %d: %Rrc\n", pVCpu->pgm.s.enmGuestMode, rc));
3321	return rc;
3322	}
3323	}
3324
3325	/*
3326	* Load new paging mode data.
3327	*/
3328	pgmR3ModeDataSwitch(pVM, pVCpu, enmShadowMode, enmGuestMode);
3329
3330	/*
3331	* Enter new shadow mode (if changed).
3332	*/
3333	if ( enmShadowMode != pVCpu->pgm.s.enmShadowMode
3334	\|\| fForceShwEnterExit)
3335	{
3336	int rc;
3337	pVCpu->pgm.s.enmShadowMode = enmShadowMode;
3338	switch (enmShadowMode)
3339	{
3340	case PGMMODE_32_BIT:
3341	rc = PGM_SHW_NAME_32BIT(Enter)(pVCpu, false);
3342	break;
3343	case PGMMODE_PAE:
3344	case PGMMODE_PAE_NX:
3345	rc = PGM_SHW_NAME_PAE(Enter)(pVCpu, false);
3346	break;
3347	case PGMMODE_AMD64:
3348	case PGMMODE_AMD64_NX:
3349	rc = PGM_SHW_NAME_AMD64(Enter)(pVCpu, fIsNewGuestPagingMode64Bits);
3350	break;
3351	case PGMMODE_NESTED:
3352	rc = PGM_SHW_NAME_NESTED(Enter)(pVCpu, fIsNewGuestPagingMode64Bits);
3353	break;
3354	case PGMMODE_EPT:
3355	rc = PGM_SHW_NAME_EPT(Enter)(pVCpu, fIsNewGuestPagingMode64Bits);
3356	break;
3357	case PGMMODE_REAL:
3358	case PGMMODE_PROTECTED:
3359	default:
3360	AssertReleaseMsgFailed(("enmShadowMode=%d\n", enmShadowMode));
3361	return VERR_INTERNAL_ERROR;
3362	}
3363	if (RT_FAILURE(rc))
3364	{
3365	AssertReleaseMsgFailed(("Entering enmShadowMode=%d failed: %Rrc\n", enmShadowMode, rc));
3366	pVCpu->pgm.s.enmShadowMode = PGMMODE_INVALID;
3367	return rc;
3368	}
3369	}
3370
3371	/*
3372	* Always flag the necessary updates
3373	*/
3374	VMCPU_FF_SET(pVCpu, VMCPU_FF_PGM_SYNC_CR3);
3375
3376	/*
3377	* Enter the new guest and shadow+guest modes.
3378	*/
3379	int rc = -1;
3380	int rc2 = -1;
3381	RTGCPHYS GCPhysCR3 = NIL_RTGCPHYS;
3382	pVCpu->pgm.s.enmGuestMode = enmGuestMode;
3383	switch (enmGuestMode)
3384	{
3385	case PGMMODE_REAL:
3386	rc = PGM_GST_NAME_REAL(Enter)(pVCpu, NIL_RTGCPHYS);
3387	switch (pVCpu->pgm.s.enmShadowMode)
3388	{
3389	case PGMMODE_32_BIT:
3390	rc2 = PGM_BTH_NAME_32BIT_REAL(Enter)(pVCpu, NIL_RTGCPHYS);
3391	break;
3392	case PGMMODE_PAE:
3393	case PGMMODE_PAE_NX:
3394	rc2 = PGM_BTH_NAME_PAE_REAL(Enter)(pVCpu, NIL_RTGCPHYS);
3395	break;
3396	case PGMMODE_NESTED:
3397	rc2 = PGM_BTH_NAME_NESTED_REAL(Enter)(pVCpu, NIL_RTGCPHYS);
3398	break;
3399	case PGMMODE_EPT:
3400	rc2 = PGM_BTH_NAME_EPT_REAL(Enter)(pVCpu, NIL_RTGCPHYS);
3401	break;
3402	case PGMMODE_AMD64:
3403	case PGMMODE_AMD64_NX:
3404	AssertMsgFailed(("Should use PAE shadow mode!\n"));
3405	default: AssertFailed(); break;
3406	}
3407	break;
3408
3409	case PGMMODE_PROTECTED:
3410	rc = PGM_GST_NAME_PROT(Enter)(pVCpu, NIL_RTGCPHYS);
3411	switch (pVCpu->pgm.s.enmShadowMode)
3412	{
3413	case PGMMODE_32_BIT:
3414	rc2 = PGM_BTH_NAME_32BIT_PROT(Enter)(pVCpu, NIL_RTGCPHYS);
3415	break;
3416	case PGMMODE_PAE:
3417	case PGMMODE_PAE_NX:
3418	rc2 = PGM_BTH_NAME_PAE_PROT(Enter)(pVCpu, NIL_RTGCPHYS);
3419	break;
3420	case PGMMODE_NESTED:
3421	rc2 = PGM_BTH_NAME_NESTED_PROT(Enter)(pVCpu, NIL_RTGCPHYS);
3422	break;
3423	case PGMMODE_EPT:
3424	rc2 = PGM_BTH_NAME_EPT_PROT(Enter)(pVCpu, NIL_RTGCPHYS);
3425	break;
3426	case PGMMODE_AMD64:
3427	case PGMMODE_AMD64_NX:
3428	AssertMsgFailed(("Should use PAE shadow mode!\n"));
3429	default: AssertFailed(); break;
3430	}
3431	break;
3432
3433	case PGMMODE_32_BIT:
3434	GCPhysCR3 = CPUMGetGuestCR3(pVCpu) & X86_CR3_PAGE_MASK;
3435	rc = PGM_GST_NAME_32BIT(Enter)(pVCpu, GCPhysCR3);
3436	switch (pVCpu->pgm.s.enmShadowMode)
3437	{
3438	case PGMMODE_32_BIT:
3439	rc2 = PGM_BTH_NAME_32BIT_32BIT(Enter)(pVCpu, GCPhysCR3);
3440	break;
3441	case PGMMODE_PAE:
3442	case PGMMODE_PAE_NX:
3443	rc2 = PGM_BTH_NAME_PAE_32BIT(Enter)(pVCpu, GCPhysCR3);
3444	break;
3445	case PGMMODE_NESTED:
3446	rc2 = PGM_BTH_NAME_NESTED_32BIT(Enter)(pVCpu, GCPhysCR3);
3447	break;
3448	case PGMMODE_EPT:
3449	rc2 = PGM_BTH_NAME_EPT_32BIT(Enter)(pVCpu, GCPhysCR3);
3450	break;
3451	case PGMMODE_AMD64:
3452	case PGMMODE_AMD64_NX:
3453	AssertMsgFailed(("Should use PAE shadow mode!\n"));
3454	default: AssertFailed(); break;
3455	}
3456	break;
3457
3458	case PGMMODE_PAE_NX:
3459	case PGMMODE_PAE:
3460	{
3461	uint32_t u32Dummy, u32Features;
3462
3463	CPUMGetGuestCpuId(pVCpu, 1, &u32Dummy, &u32Dummy, &u32Dummy, &u32Features);
3464	if (!(u32Features & X86_CPUID_FEATURE_EDX_PAE))
3465	return VMSetRuntimeError(pVM, VMSETRTERR_FLAGS_FATAL, "PAEmode",
3466	N_("The guest is trying to switch to the PAE mode which is currently disabled by default in VirtualBox. PAE support can be enabled using the VM settings (General/Advanced)"));
3467
3468	GCPhysCR3 = CPUMGetGuestCR3(pVCpu) & X86_CR3_PAE_PAGE_MASK;
3469	rc = PGM_GST_NAME_PAE(Enter)(pVCpu, GCPhysCR3);
3470	switch (pVCpu->pgm.s.enmShadowMode)
3471	{
3472	case PGMMODE_PAE:
3473	case PGMMODE_PAE_NX:
3474	rc2 = PGM_BTH_NAME_PAE_PAE(Enter)(pVCpu, GCPhysCR3);
3475	break;
3476	case PGMMODE_NESTED:
3477	rc2 = PGM_BTH_NAME_NESTED_PAE(Enter)(pVCpu, GCPhysCR3);
3478	break;
3479	case PGMMODE_EPT:
3480	rc2 = PGM_BTH_NAME_EPT_PAE(Enter)(pVCpu, GCPhysCR3);
3481	break;
3482	case PGMMODE_32_BIT:
3483	case PGMMODE_AMD64:
3484	case PGMMODE_AMD64_NX:
3485	AssertMsgFailed(("Should use PAE shadow mode!\n"));
3486	default: AssertFailed(); break;
3487	}
3488	break;
3489	}
3490
3491	#ifdef VBOX_WITH_64_BITS_GUESTS
3492	case PGMMODE_AMD64_NX:
3493	case PGMMODE_AMD64:
3494	GCPhysCR3 = CPUMGetGuestCR3(pVCpu) & UINT64_C(0xfffffffffffff000); /** @todo define this mask! */
3495	rc = PGM_GST_NAME_AMD64(Enter)(pVCpu, GCPhysCR3);
3496	switch (pVCpu->pgm.s.enmShadowMode)
3497	{
3498	case PGMMODE_AMD64:
3499	case PGMMODE_AMD64_NX:
3500	rc2 = PGM_BTH_NAME_AMD64_AMD64(Enter)(pVCpu, GCPhysCR3);
3501	break;
3502	case PGMMODE_NESTED:
3503	rc2 = PGM_BTH_NAME_NESTED_AMD64(Enter)(pVCpu, GCPhysCR3);
3504	break;
3505	case PGMMODE_EPT:
3506	rc2 = PGM_BTH_NAME_EPT_AMD64(Enter)(pVCpu, GCPhysCR3);
3507	break;
3508	case PGMMODE_32_BIT:
3509	case PGMMODE_PAE:
3510	case PGMMODE_PAE_NX:
3511	AssertMsgFailed(("Should use AMD64 shadow mode!\n"));
3512	default: AssertFailed(); break;
3513	}
3514	break;
3515	#endif
3516
3517	default:
3518	AssertReleaseMsgFailed(("enmGuestMode=%d\n", enmGuestMode));
3519	rc = VERR_NOT_IMPLEMENTED;
3520	break;
3521	}
3522
3523	/* status codes. */
3524	AssertRC(rc);
3525	AssertRC(rc2);
3526	if (RT_SUCCESS(rc))
3527	{
3528	rc = rc2;
3529	if (RT_SUCCESS(rc)) /* no informational status codes. */
3530	rc = VINF_SUCCESS;
3531	}
3532
3533	/* Notify HWACCM as well. */
3534	HWACCMR3PagingModeChanged(pVM, pVCpu, pVCpu->pgm.s.enmShadowMode, pVCpu->pgm.s.enmGuestMode);
3535	return rc;
3536	}
3537
3538
3539	/**
3540	* Called by pgmPoolFlushAllInt prior to flushing the pool.
3541	*
3542	* @returns VBox status code, fully asserted.
3543	* @param pVM The VM handle.
3544	* @param pVCpu The VMCPU to operate on.
3545	*/
3546	int pgmR3ExitShadowModeBeforePoolFlush(PVM pVM, PVMCPU pVCpu)
3547	{
3548	/* Unmap the old CR3 value before flushing everything. */
3549	int rc = PGM_BTH_PFN(UnmapCR3, pVCpu)(pVCpu);
3550	AssertRC(rc);
3551
3552	/* Exit the current shadow paging mode as well; nested paging and EPT use a root CR3 which will get flushed here. */
3553	rc = PGM_SHW_PFN(Exit, pVCpu)(pVCpu);
3554	AssertRC(rc);
3555	Assert(pVCpu->pgm.s.pShwPageCR3R3 == NULL);
3556	return rc;
3557	}
3558
3559
3560	/**
3561	* Called by pgmPoolFlushAllInt after flushing the pool.
3562	*
3563	* @returns VBox status code, fully asserted.
3564	* @param pVM The VM handle.
3565	* @param pVCpu The VMCPU to operate on.
3566	*/
3567	int pgmR3ReEnterShadowModeAfterPoolFlush(PVM pVM, PVMCPU pVCpu)
3568	{
3569	pVCpu->pgm.s.enmShadowMode = PGMMODE_INVALID;
3570	int rc = PGMR3ChangeMode(pVM, pVCpu, PGMGetGuestMode(pVCpu));
3571	Assert(VMCPU_FF_ISSET(pVCpu, VMCPU_FF_PGM_SYNC_CR3));
3572	AssertRCReturn(rc, rc);
3573	AssertRCSuccessReturn(rc, VERR_IPE_UNEXPECTED_INFO_STATUS);
3574
3575	Assert(pVCpu->pgm.s.pShwPageCR3R3 != NULL);
3576	AssertMsg( pVCpu->pgm.s.enmShadowMode >= PGMMODE_NESTED
3577	\|\| CPUMGetHyperCR3(pVCpu) == PGMGetHyperCR3(pVCpu),
3578	("%RHp != %RHp %s\n", (RTHCPHYS)CPUMGetHyperCR3(pVCpu), PGMGetHyperCR3(pVCpu), PGMGetModeName(pVCpu->pgm.s.enmShadowMode)));
3579	return rc;
3580	}
3581
3582	#ifdef VBOX_WITH_DEBUGGER
3583
3584	/**
3585	* The '.pgmram' command.
3586	*
3587	* @returns VBox status.
3588	* @param pCmd Pointer to the command descriptor (as registered).
3589	* @param pCmdHlp Pointer to command helper functions.
3590	* @param pVM Pointer to the current VM (if any).
3591	* @param paArgs Pointer to (readonly) array of arguments.
3592	* @param cArgs Number of arguments in the array.
3593	*/
3594	static DECLCALLBACK(int) pgmR3CmdRam(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs)
3595	{
3596	/*
3597	* Validate input.
3598	*/
3599	if (!pVM)
3600	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires a VM to be selected.\n");
3601	if (!pVM->pgm.s.pRamRangesXR3)
3602	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Sorry, no Ram is registered.\n");
3603
3604	/*
3605	* Dump the ranges.
3606	*/
3607	int rc = pCmdHlp->pfnPrintf(pCmdHlp, NULL, "From - To (incl) pvHC\n");
3608	PPGMRAMRANGE pRam;
3609	for (pRam = pVM->pgm.s.pRamRangesXR3; pRam; pRam = pRam->pNextR3)
3610	{
3611	rc = pCmdHlp->pfnPrintf(pCmdHlp, NULL,
3612	"%RGp - %RGp %p\n",
3613	pRam->GCPhys, pRam->GCPhysLast, pRam->pvR3);
3614	if (RT_FAILURE(rc))
3615	return rc;
3616	}
3617
3618	return VINF_SUCCESS;
3619	}
3620
3621
3622	/**
3623	* The '.pgmerror' and '.pgmerroroff' commands.
3624	*
3625	* @returns VBox status.
3626	* @param pCmd Pointer to the command descriptor (as registered).
3627	* @param pCmdHlp Pointer to command helper functions.
3628	* @param pVM Pointer to the current VM (if any).
3629	* @param paArgs Pointer to (readonly) array of arguments.
3630	* @param cArgs Number of arguments in the array.
3631	*/
3632	static DECLCALLBACK(int) pgmR3CmdError(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs)
3633	{
3634	/*
3635	* Validate input.
3636	*/
3637	if (!pVM)
3638	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires a VM to be selected.\n");
3639	AssertReturn(cArgs == 0 \|\| (cArgs == 1 && paArgs[0].enmType == DBGCVAR_TYPE_STRING),
3640	pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: Hit bug in the parser.\n"));
3641
3642	if (!cArgs)
3643	{
3644	/*
3645	* Print the list of error injection locations with status.
3646	*/
3647	pCmdHlp->pfnPrintf(pCmdHlp, NULL, "PGM error inject locations:\n");
3648	pCmdHlp->pfnPrintf(pCmdHlp, NULL, " handy - %RTbool\n", pVM->pgm.s.fErrInjHandyPages);
3649	}
3650	else
3651	{
3652
3653	/*
3654	* String switch on where to inject the error.
3655	*/
3656	bool const fNewState = !strcmp(pCmd->pszCmd, "pgmerror");
3657	const char *pszWhere = paArgs[0].u.pszString;
3658	if (!strcmp(pszWhere, "handy"))
3659	ASMAtomicWriteBool(&pVM->pgm.s.fErrInjHandyPages, fNewState);
3660	else
3661	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: Invalid 'where' value: %s.\n", pszWhere);
3662	pCmdHlp->pfnPrintf(pCmdHlp, NULL, "done\n");
3663	}
3664	return VINF_SUCCESS;
3665	}
3666
3667
3668	/**
3669	* The '.pgmsync' command.
3670	*
3671	* @returns VBox status.
3672	* @param pCmd Pointer to the command descriptor (as registered).
3673	* @param pCmdHlp Pointer to command helper functions.
3674	* @param pVM Pointer to the current VM (if any).
3675	* @param paArgs Pointer to (readonly) array of arguments.
3676	* @param cArgs Number of arguments in the array.
3677	*/
3678	static DECLCALLBACK(int) pgmR3CmdSync(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs)
3679	{
3680	/** @todo SMP support */
3681
3682	/*
3683	* Validate input.
3684	*/
3685	if (!pVM)
3686	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires a VM to be selected.\n");
3687
3688	PVMCPU pVCpu = &pVM->aCpus[0];
3689
3690	/*
3691	* Force page directory sync.
3692	*/
3693	VMCPU_FF_SET(pVCpu, VMCPU_FF_PGM_SYNC_CR3);
3694
3695	int rc = pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Forcing page directory sync.\n");
3696	if (RT_FAILURE(rc))
3697	return rc;
3698
3699	return VINF_SUCCESS;
3700	}
3701
3702
3703	#ifdef VBOX_STRICT
3704	/**
3705	* The '.pgmassertcr3' command.
3706	*
3707	* @returns VBox status.
3708	* @param pCmd Pointer to the command descriptor (as registered).
3709	* @param pCmdHlp Pointer to command helper functions.
3710	* @param pVM Pointer to the current VM (if any).
3711	* @param paArgs Pointer to (readonly) array of arguments.
3712	* @param cArgs Number of arguments in the array.
3713	*/
3714	static DECLCALLBACK(int) pgmR3CmdAssertCR3(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs)
3715	{
3716	/** @todo SMP support!! */
3717
3718	/*
3719	* Validate input.
3720	*/
3721	if (!pVM)
3722	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires a VM to be selected.\n");
3723
3724	PVMCPU pVCpu = &pVM->aCpus[0];
3725
3726	int rc = pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Checking shadow CR3 page tables for consistency.\n");
3727	if (RT_FAILURE(rc))
3728	return rc;
3729
3730	PGMAssertCR3(pVM, pVCpu, CPUMGetGuestCR3(pVCpu), CPUMGetGuestCR4(pVCpu));
3731
3732	return VINF_SUCCESS;
3733	}
3734	#endif /* VBOX_STRICT */
3735
3736
3737	/**
3738	* The '.pgmsyncalways' command.
3739	*
3740	* @returns VBox status.
3741	* @param pCmd Pointer to the command descriptor (as registered).
3742	* @param pCmdHlp Pointer to command helper functions.
3743	* @param pVM Pointer to the current VM (if any).
3744	* @param paArgs Pointer to (readonly) array of arguments.
3745	* @param cArgs Number of arguments in the array.
3746	*/
3747	static DECLCALLBACK(int) pgmR3CmdSyncAlways(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs)
3748	{
3749	/** @todo SMP support!! */
3750	PVMCPU pVCpu = &pVM->aCpus[0];
3751
3752	/*
3753	* Validate input.
3754	*/
3755	if (!pVM)
3756	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires a VM to be selected.\n");
3757
3758	/*
3759	* Force page directory sync.
3760	*/
3761	if (pVCpu->pgm.s.fSyncFlags & PGM_SYNC_ALWAYS)
3762	{
3763	ASMAtomicAndU32(&pVCpu->pgm.s.fSyncFlags, ~PGM_SYNC_ALWAYS);
3764	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Disabled permanent forced page directory syncing.\n");
3765	}
3766	else
3767	{
3768	ASMAtomicOrU32(&pVCpu->pgm.s.fSyncFlags, PGM_SYNC_ALWAYS);
3769	VMCPU_FF_SET(pVCpu, VMCPU_FF_PGM_SYNC_CR3);
3770	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Enabled permanent forced page directory syncing.\n");
3771	}
3772	}
3773
3774
3775	/**
3776	* The '.pgmphystofile' command.
3777	*
3778	* @returns VBox status.
3779	* @param pCmd Pointer to the command descriptor (as registered).
3780	* @param pCmdHlp Pointer to command helper functions.
3781	* @param pVM Pointer to the current VM (if any).
3782	* @param paArgs Pointer to (readonly) array of arguments.
3783	* @param cArgs Number of arguments in the array.
3784	*/
3785	static DECLCALLBACK(int) pgmR3CmdPhysToFile(PCDBGCCMD pCmd, PDBGCCMDHLP pCmdHlp, PVM pVM, PCDBGCVAR paArgs, unsigned cArgs)
3786	{
3787	/*
3788	* Validate input.
3789	*/
3790	if (!pVM)
3791	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: The command requires a VM to be selected.\n");
3792	if ( cArgs < 1
3793	\|\| cArgs > 2
3794	\|\| paArgs[0].enmType != DBGCVAR_TYPE_STRING
3795	\|\| ( cArgs > 1
3796	&& paArgs[1].enmType != DBGCVAR_TYPE_STRING))
3797	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: parser error, invalid arguments.\n");
3798	if ( cArgs >= 2
3799	&& strcmp(paArgs[1].u.pszString, "nozero"))
3800	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: Invalid 2nd argument '%s', must be 'nozero'.\n", paArgs[1].u.pszString);
3801	bool fIncZeroPgs = cArgs < 2;
3802
3803	/*
3804	* Open the output file and get the ram parameters.
3805	*/
3806	RTFILE hFile;
3807	int rc = RTFileOpen(&hFile, paArgs[0].u.pszString, RTFILE_O_WRITE \| RTFILE_O_CREATE_REPLACE \| RTFILE_O_DENY_WRITE);
3808	if (RT_FAILURE(rc))
3809	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: RTFileOpen(,'%s',) -> %Rrc.\n", paArgs[0].u.pszString, rc);
3810
3811	uint32_t cbRamHole = 0;
3812	CFGMR3QueryU32Def(CFGMR3GetRoot(pVM), "RamHoleSize", &cbRamHole, MM_RAM_HOLE_SIZE_DEFAULT);
3813	uint64_t cbRam = 0;
3814	CFGMR3QueryU64Def(CFGMR3GetRoot(pVM), "RamSize", &cbRam, 0);
3815	RTGCPHYS GCPhysEnd = cbRam + cbRamHole;
3816
3817	/*
3818	* Dump the physical memory, page by page.
3819	*/
3820	RTGCPHYS GCPhys = 0;
3821	char abZeroPg[PAGE_SIZE];
3822	RT_ZERO(abZeroPg);
3823
3824	pgmLock(pVM);
3825	for (PPGMRAMRANGE pRam = pVM->pgm.s.pRamRangesXR3;
3826	pRam && pRam->GCPhys < GCPhysEnd && RT_SUCCESS(rc);
3827	pRam = pRam->pNextR3)
3828	{
3829	/* fill the gap */
3830	if (pRam->GCPhys > GCPhys && fIncZeroPgs)
3831	{
3832	while (pRam->GCPhys > GCPhys && RT_SUCCESS(rc))
3833	{
3834	rc = RTFileWrite(hFile, abZeroPg, PAGE_SIZE, NULL);
3835	GCPhys += PAGE_SIZE;
3836	}
3837	}
3838
3839	PCPGMPAGE pPage = &pRam->aPages[0];
3840	while (GCPhys < pRam->GCPhysLast && RT_SUCCESS(rc))
3841	{
3842	if ( PGM_PAGE_IS_ZERO(pPage)
3843	\|\| PGM_PAGE_IS_BALLOONED(pPage))
3844	{
3845	if (fIncZeroPgs)
3846	{
3847	rc = RTFileWrite(hFile, abZeroPg, PAGE_SIZE, NULL);
3848	if (RT_FAILURE(rc))
3849	pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: RTFileWrite -> %Rrc at GCPhys=%RGp.\n", rc, GCPhys);
3850	}
3851	}
3852	else
3853	{
3854	switch (PGM_PAGE_GET_TYPE(pPage))
3855	{
3856	case PGMPAGETYPE_RAM:
3857	case PGMPAGETYPE_ROM_SHADOW: /* trouble?? */
3858	case PGMPAGETYPE_ROM:
3859	case PGMPAGETYPE_MMIO2:
3860	{
3861	void const *pvPage;
3862	PGMPAGEMAPLOCK Lock;
3863	rc = PGMPhysGCPhys2CCPtrReadOnly(pVM, GCPhys, &pvPage, &Lock);
3864	if (RT_SUCCESS(rc))
3865	{
3866	rc = RTFileWrite(hFile, pvPage, PAGE_SIZE, NULL);
3867	PGMPhysReleasePageMappingLock(pVM, &Lock);
3868	if (RT_FAILURE(rc))
3869	pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: RTFileWrite -> %Rrc at GCPhys=%RGp.\n", rc, GCPhys);
3870	}
3871	else
3872	pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: PGMPhysGCPhys2CCPtrReadOnly -> %Rrc at GCPhys=%RGp.\n", rc, GCPhys);
3873	break;
3874	}
3875
3876	default:
3877	AssertFailed();
3878	case PGMPAGETYPE_MMIO2_ALIAS_MMIO:
3879	case PGMPAGETYPE_MMIO:
3880	if (fIncZeroPgs)
3881	{
3882	rc = RTFileWrite(hFile, abZeroPg, PAGE_SIZE, NULL);
3883	if (RT_FAILURE(rc))
3884	pCmdHlp->pfnPrintf(pCmdHlp, NULL, "error: RTFileWrite -> %Rrc at GCPhys=%RGp.\n", rc, GCPhys);
3885	}
3886	break;
3887	}
3888	}
3889
3890
3891	/* advance */
3892	GCPhys += PAGE_SIZE;
3893	pPage++;
3894	}
3895	}
3896	pgmUnlock(pVM);
3897
3898	RTFileClose(hFile);
3899	if (RT_SUCCESS(rc))
3900	return pCmdHlp->pfnPrintf(pCmdHlp, NULL, "Successfully saved physical memory to '%s'.\n", paArgs[0].u.pszString);
3901	return VINF_SUCCESS;
3902	}
3903
3904	#endif /* VBOX_WITH_DEBUGGER */
3905
3906	/**
3907	* pvUser argument of the pgmR3CheckIntegrity*Node callbacks.
3908	*/
3909	typedef struct PGMCHECKINTARGS
3910	{
3911	bool fLeftToRight; /*< true: left-to-right; false: right-to-left. /
3912	PPGMPHYSHANDLER pPrevPhys;
3913	PPGMVIRTHANDLER pPrevVirt;
3914	PPGMPHYS2VIRTHANDLER pPrevPhys2Virt;
3915	PVM pVM;
3916	} PGMCHECKINTARGS, *PPGMCHECKINTARGS;
3917
3918	/**
3919	* Validate a node in the physical handler tree.
3920	*
3921	* @returns 0 on if ok, other wise 1.
3922	* @param pNode The handler node.
3923	* @param pvUser pVM.
3924	*/
3925	static DECLCALLBACK(int) pgmR3CheckIntegrityPhysHandlerNode(PAVLROGCPHYSNODECORE pNode, void *pvUser)
3926	{
3927	PPGMCHECKINTARGS pArgs = (PPGMCHECKINTARGS)pvUser;
3928	PPGMPHYSHANDLER pCur = (PPGMPHYSHANDLER)pNode;
3929	AssertReleaseReturn(!((uintptr_t)pCur & 7), 1);
3930	AssertReleaseMsg(pCur->Core.Key <= pCur->Core.KeyLast,("pCur=%p %RGp-%RGp %s\n", pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc));
3931	AssertReleaseMsg( !pArgs->pPrevPhys
3932	\|\| (pArgs->fLeftToRight ? pArgs->pPrevPhys->Core.KeyLast < pCur->Core.Key : pArgs->pPrevPhys->Core.KeyLast > pCur->Core.Key),
3933	("pPrevPhys=%p %RGp-%RGp %s\n"
3934	" pCur=%p %RGp-%RGp %s\n",
3935	pArgs->pPrevPhys, pArgs->pPrevPhys->Core.Key, pArgs->pPrevPhys->Core.KeyLast, pArgs->pPrevPhys->pszDesc,
3936	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc));
3937	pArgs->pPrevPhys = pCur;
3938	return 0;
3939	}
3940
3941
3942	/**
3943	* Validate a node in the virtual handler tree.
3944	*
3945	* @returns 0 on if ok, other wise 1.
3946	* @param pNode The handler node.
3947	* @param pvUser pVM.
3948	*/
3949	static DECLCALLBACK(int) pgmR3CheckIntegrityVirtHandlerNode(PAVLROGCPTRNODECORE pNode, void *pvUser)
3950	{
3951	PPGMCHECKINTARGS pArgs = (PPGMCHECKINTARGS)pvUser;
3952	PPGMVIRTHANDLER pCur = (PPGMVIRTHANDLER)pNode;
3953	AssertReleaseReturn(!((uintptr_t)pCur & 7), 1);
3954	AssertReleaseMsg(pCur->Core.Key <= pCur->Core.KeyLast,("pCur=%p %RGv-%RGv %s\n", pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc));
3955	AssertReleaseMsg( !pArgs->pPrevVirt
3956	\|\| (pArgs->fLeftToRight ? pArgs->pPrevVirt->Core.KeyLast < pCur->Core.Key : pArgs->pPrevVirt->Core.KeyLast > pCur->Core.Key),
3957	("pPrevVirt=%p %RGv-%RGv %s\n"
3958	" pCur=%p %RGv-%RGv %s\n",
3959	pArgs->pPrevVirt, pArgs->pPrevVirt->Core.Key, pArgs->pPrevVirt->Core.KeyLast, pArgs->pPrevVirt->pszDesc,
3960	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc));
3961	for (unsigned iPage = 0; iPage < pCur->cPages; iPage++)
3962	{
3963	AssertReleaseMsg(pCur->aPhysToVirt[iPage].offVirtHandler == -RT_OFFSETOF(PGMVIRTHANDLER, aPhysToVirt[iPage]),
3964	("pCur=%p %RGv-%RGv %s\n"
3965	"iPage=%d offVirtHandle=%#x expected %#x\n",
3966	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->pszDesc,
3967	iPage, pCur->aPhysToVirt[iPage].offVirtHandler, -RT_OFFSETOF(PGMVIRTHANDLER, aPhysToVirt[iPage])));
3968	}
3969	pArgs->pPrevVirt = pCur;
3970	return 0;
3971	}
3972
3973
3974	/**
3975	* Validate a node in the virtual handler tree.
3976	*
3977	* @returns 0 on if ok, other wise 1.
3978	* @param pNode The handler node.
3979	* @param pvUser pVM.
3980	*/
3981	static DECLCALLBACK(int) pgmR3CheckIntegrityPhysToVirtHandlerNode(PAVLROGCPHYSNODECORE pNode, void *pvUser)
3982	{
3983	PPGMCHECKINTARGS pArgs = (PPGMCHECKINTARGS)pvUser;
3984	PPGMPHYS2VIRTHANDLER pCur = (PPGMPHYS2VIRTHANDLER)pNode;
3985	AssertReleaseMsgReturn(!((uintptr_t)pCur & 3), ("\n"), 1);
3986	AssertReleaseMsgReturn(!(pCur->offVirtHandler & 3), ("\n"), 1);
3987	AssertReleaseMsg(pCur->Core.Key <= pCur->Core.KeyLast,("pCur=%p %RGp-%RGp\n", pCur, pCur->Core.Key, pCur->Core.KeyLast));
3988	AssertReleaseMsg( !pArgs->pPrevPhys2Virt
3989	\|\| (pArgs->fLeftToRight ? pArgs->pPrevPhys2Virt->Core.KeyLast < pCur->Core.Key : pArgs->pPrevPhys2Virt->Core.KeyLast > pCur->Core.Key),
3990	("pPrevPhys2Virt=%p %RGp-%RGp\n"
3991	" pCur=%p %RGp-%RGp\n",
3992	pArgs->pPrevPhys2Virt, pArgs->pPrevPhys2Virt->Core.Key, pArgs->pPrevPhys2Virt->Core.KeyLast,
3993	pCur, pCur->Core.Key, pCur->Core.KeyLast));
3994	AssertReleaseMsg( !pArgs->pPrevPhys2Virt
3995	\|\| (pArgs->fLeftToRight ? pArgs->pPrevPhys2Virt->Core.KeyLast < pCur->Core.Key : pArgs->pPrevPhys2Virt->Core.KeyLast > pCur->Core.Key),
3996	("pPrevPhys2Virt=%p %RGp-%RGp\n"
3997	" pCur=%p %RGp-%RGp\n",
3998	pArgs->pPrevPhys2Virt, pArgs->pPrevPhys2Virt->Core.Key, pArgs->pPrevPhys2Virt->Core.KeyLast,
3999	pCur, pCur->Core.Key, pCur->Core.KeyLast));
4000	AssertReleaseMsg((pCur->offNextAlias & (PGMPHYS2VIRTHANDLER_IN_TREE \| PGMPHYS2VIRTHANDLER_IS_HEAD)) == (PGMPHYS2VIRTHANDLER_IN_TREE \| PGMPHYS2VIRTHANDLER_IS_HEAD),
4001	("pCur=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
4002	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias));
4003	if (pCur->offNextAlias & PGMPHYS2VIRTHANDLER_OFF_MASK)
4004	{
4005	PPGMPHYS2VIRTHANDLER pCur2 = pCur;
4006	for (;;)
4007	{
4008	pCur2 = (PPGMPHYS2VIRTHANDLER)((intptr_t)pCur + (pCur->offNextAlias & PGMPHYS2VIRTHANDLER_OFF_MASK));
4009	AssertReleaseMsg(pCur2 != pCur,
4010	(" pCur=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
4011	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias));
4012	AssertReleaseMsg((pCur2->offNextAlias & (PGMPHYS2VIRTHANDLER_IN_TREE \| PGMPHYS2VIRTHANDLER_IS_HEAD)) == PGMPHYS2VIRTHANDLER_IN_TREE,
4013	(" pCur=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n"
4014	"pCur2=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
4015	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias,
4016	pCur2, pCur2->Core.Key, pCur2->Core.KeyLast, pCur2->offVirtHandler, pCur2->offNextAlias));
4017	AssertReleaseMsg((pCur2->Core.Key ^ pCur->Core.Key) < PAGE_SIZE,
4018	(" pCur=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n"
4019	"pCur2=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
4020	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias,
4021	pCur2, pCur2->Core.Key, pCur2->Core.KeyLast, pCur2->offVirtHandler, pCur2->offNextAlias));
4022	AssertReleaseMsg((pCur2->Core.KeyLast ^ pCur->Core.KeyLast) < PAGE_SIZE,
4023	(" pCur=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n"
4024	"pCur2=%p:{.Core.Key=%RGp, .Core.KeyLast=%RGp, .offVirtHandler=%#RX32, .offNextAlias=%#RX32}\n",
4025	pCur, pCur->Core.Key, pCur->Core.KeyLast, pCur->offVirtHandler, pCur->offNextAlias,
4026	pCur2, pCur2->Core.Key, pCur2->Core.KeyLast, pCur2->offVirtHandler, pCur2->offNextAlias));
4027	if (!(pCur2->offNextAlias & PGMPHYS2VIRTHANDLER_OFF_MASK))
4028	break;
4029	}
4030	}
4031
4032	pArgs->pPrevPhys2Virt = pCur;
4033	return 0;
4034	}
4035
4036
4037	/**
4038	* Perform an integrity check on the PGM component.
4039	*
4040	* @returns VINF_SUCCESS if everything is fine.
4041	* @returns VBox error status after asserting on integrity breach.
4042	* @param pVM The VM handle.
4043	*/
4044	VMMR3DECL(int) PGMR3CheckIntegrity(PVM pVM)
4045	{
4046	AssertReleaseReturn(pVM->pgm.s.offVM, VERR_INTERNAL_ERROR);
4047
4048	/*
4049	* Check the trees.
4050	*/
4051	int cErrors = 0;
4052	const static PGMCHECKINTARGS s_LeftToRight = { true, NULL, NULL, NULL, pVM };
4053	const static PGMCHECKINTARGS s_RightToLeft = { false, NULL, NULL, NULL, pVM };
4054	PGMCHECKINTARGS Args = s_LeftToRight;
4055	cErrors += RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesR3->PhysHandlers, true, pgmR3CheckIntegrityPhysHandlerNode, &Args);
4056	Args = s_RightToLeft;
4057	cErrors += RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesR3->PhysHandlers, false, pgmR3CheckIntegrityPhysHandlerNode, &Args);
4058	Args = s_LeftToRight;
4059	cErrors += RTAvlroGCPtrDoWithAll( &pVM->pgm.s.pTreesR3->VirtHandlers, true, pgmR3CheckIntegrityVirtHandlerNode, &Args);
4060	Args = s_RightToLeft;
4061	cErrors += RTAvlroGCPtrDoWithAll( &pVM->pgm.s.pTreesR3->VirtHandlers, false, pgmR3CheckIntegrityVirtHandlerNode, &Args);
4062	Args = s_LeftToRight;
4063	cErrors += RTAvlroGCPtrDoWithAll( &pVM->pgm.s.pTreesR3->HyperVirtHandlers, true, pgmR3CheckIntegrityVirtHandlerNode, &Args);
4064	Args = s_RightToLeft;
4065	cErrors += RTAvlroGCPtrDoWithAll( &pVM->pgm.s.pTreesR3->HyperVirtHandlers, false, pgmR3CheckIntegrityVirtHandlerNode, &Args);
4066	Args = s_LeftToRight;
4067	cErrors += RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesR3->PhysToVirtHandlers, true, pgmR3CheckIntegrityPhysToVirtHandlerNode, &Args);
4068	Args = s_RightToLeft;
4069	cErrors += RTAvlroGCPhysDoWithAll(&pVM->pgm.s.pTreesR3->PhysToVirtHandlers, false, pgmR3CheckIntegrityPhysToVirtHandlerNode, &Args);
4070
4071	return !cErrors ? VINF_SUCCESS : VERR_INTERNAL_ERROR;
4072	}
4073

Note: See TracBrowser for help on using the repository browser.

source: vbox/trunk/src/VBox/VMM/VMMR3/PGM.cpp@ 37423

Download in other formats: