Subject: mm: page allocator: Update free page counters after pages are placed on the free list

From: Mel Gorman <mel@csn.ul.ie>

When allocating a page, the system uses NR_FREE_PAGES counters to determine
in advance if watermarks are being met. This check is made without interrupts
disabled or the zone lock held.

Unfortunately, when pages are being freed in batch, the counters are updated
before the pages are added on the list. During this window, the counters are
misleading as the pages do not exist yet. When under significant pressure
on systems with large numbers of CPUs, it's possible for processes to make
progress even though they should have been stalled. This is particularly
pragmetic if a number of the processes are using GFP_ATOMIC as the min
watermark can be accidentally breached and in extreme cases, the system
can livelock.

This patch updates the counters after the pages have been added to the
list. This makes the allocator more cautious with respect to preserving
the watermarks and mitigates livelock possibilities.

Signed-off-by: Mel Gorman <mel@csn.ul.ie>
Signed-off-by: Andrea Arcangeli <aarcange@redhat.com>
---

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -589,12 +589,12 @@ static void free_pcppages_bulk(struct zo
 {
 	int migratetype = 0;
 	int batch_free = 0;
+	int freed = count;
 
 	spin_lock(&zone->lock);
 	zone->all_unreclaimable = 0;
 	zone->pages_scanned = 0;
 
-	__mod_zone_page_state(zone, NR_FREE_PAGES, count);
 	while (count) {
 		struct page *page;
 		struct list_head *list;
@@ -622,6 +622,7 @@ static void free_pcppages_bulk(struct zo
 			trace_mm_page_pcpu_drain(page, 0, page_private(page));
 		} while (--count && --batch_free && !list_empty(list));
 	}
+	__mod_zone_page_state(zone, NR_FREE_PAGES, freed);
 	spin_unlock(&zone->lock);
 }
 
@@ -632,8 +633,8 @@ static void free_one_page(struct zone *z
 	zone->all_unreclaimable = 0;
 	zone->pages_scanned = 0;
 
+	__free_one_page(page, zone, order, migratetype);
 	__mod_zone_page_state(zone, NR_FREE_PAGES, 1 << order);
-	__free_one_page(page, zone, order, migratetype);
 	spin_unlock(&zone->lock);
 }